Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4
aarch64: โ LTP
x86_64: ๐ฅ xfstests - ext4
We hope that these logs can help you find the problem quickly. For the full detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we ran or if you have any suggestions on how to make future tests more effective.
,-. ,-. ( C ) ( K ) Continuous `-',-.`-' Kernel ( I ) Integration `-' ______________________________________________________________________________
Compile testing ---------------
We compiled the kernel for 4 architectures:
aarch64: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
ppc64le: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
s390x: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
x86_64: make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
Hardware testing ---------------- We booted each kernel and ran the following tests:
aarch64: Host 1: โ Boot test โ Podman system integration test - as root โ Podman system integration test - as user โ LTP โ Loopdev Sanity โ Memory function: memfd_create โ AMTU (Abstract Machine Test Utility) โ Networking bridge: sanity โ Ethernet drivers sanity โ Networking MACsec: sanity โ Networking socket: fuzz โ Networking sctp-auth: sockopts test โ Networking: igmp conformance test โ Networking route: pmtu โ Networking route_func - local โ Networking route_func - forward โ Networking TCP: keepalive test โ Networking UDP: socket โ Networking tunnel: geneve basic test โ Networking tunnel: gre basic โ L2TP basic test โ Networking tunnel: vxlan basic โ Networking ipsec: basic netns - transport โ Networking ipsec: basic netns - tunnel โ httpd: mod_ssl smoke sanity โ tuned: tune-processes-through-perf โ ALSA PCM loopback test โ ALSA Control (mixer) Userspace Element test โ Usex - version 1.9-29 โ storage: SCSI VPD โ trace: ftrace/tracer ๐ง โ CIFS Connectathon ๐ง โ POSIX pjd-fstest suites ๐ง โ jvm - DaCapo Benchmark Suite ๐ง โ jvm - jcstress tests ๐ง โ Memory function: kaslr ๐ง โ LTP: openposix test suite ๐ง โ Networking vnic: ipvlan/basic ๐ง โ audit: audit testsuite test ๐ง โ iotop: sanity ๐ง โ storage: dm/common
Host 2:
โก Internal infrastructure issues prevented one or more tests (marked with โกโกโก) from running on this architecture. This is not the fault of the kernel that was tested.
โ Boot test โกโกโก xfstests - ext4 โกโกโก xfstests - xfs โกโกโก selinux-policy: serge-testsuite โกโกโก lvm thinp sanity โกโกโก storage: software RAID testing โกโกโก stress: stress-ng ๐ง โกโกโก IPMI driver test ๐ง โกโกโก IPMItool loop stress test ๐ง โกโกโก Storage blktests
Host 3: โ Boot test โ xfstests - ext4 โ xfstests - xfs โ selinux-policy: serge-testsuite โ lvm thinp sanity โ storage: software RAID testing ๐ง โ Storage blktests
ppc64le: Host 1: โ Boot test โ Podman system integration test - as root โ Podman system integration test - as user โ LTP โ Loopdev Sanity โ Memory function: memfd_create โ AMTU (Abstract Machine Test Utility) โ Networking bridge: sanity โ Ethernet drivers sanity โ Networking MACsec: sanity โ Networking socket: fuzz โ Networking sctp-auth: sockopts test โ Networking route: pmtu โ Networking route_func - local โ Networking route_func - forward โ Networking TCP: keepalive test โ Networking UDP: socket โ Networking tunnel: geneve basic test โ Networking tunnel: gre basic โ L2TP basic test โ Networking tunnel: vxlan basic โ Networking ipsec: basic netns - tunnel โ httpd: mod_ssl smoke sanity โ tuned: tune-processes-through-perf โ ALSA PCM loopback test โ ALSA Control (mixer) Userspace Element test โ Usex - version 1.9-29 โ trace: ftrace/tracer ๐ง โ CIFS Connectathon ๐ง โ POSIX pjd-fstest suites ๐ง โ jvm - DaCapo Benchmark Suite ๐ง โ jvm - jcstress tests ๐ง โ Memory function: kaslr ๐ง โ LTP: openposix test suite ๐ง โ Networking vnic: ipvlan/basic ๐ง โ audit: audit testsuite test ๐ง โ iotop: sanity ๐ง โ storage: dm/common
Host 2: โ Boot test ๐ฅ xfstests - ext4 โกโกโก xfstests - xfs โกโกโก selinux-policy: serge-testsuite โกโกโก lvm thinp sanity โกโกโก storage: software RAID testing ๐ง โกโกโก IPMI driver test ๐ง โกโกโก IPMItool loop stress test ๐ง โกโกโก Storage blktests
s390x: Host 1: โ Boot test โ Podman system integration test - as root โ Podman system integration test - as user โ Loopdev Sanity โ Memory function: memfd_create โ Networking bridge: sanity โ Ethernet drivers sanity โ Networking MACsec: sanity โ Networking sctp-auth: sockopts test โ Networking route: pmtu โ Networking route_func - local โ Networking route_func - forward โ Networking TCP: keepalive test โ Networking UDP: socket โ Networking tunnel: geneve basic test โ Networking tunnel: gre basic โ L2TP basic test โ Networking tunnel: vxlan basic โ Networking ipsec: basic netns - transport โ Networking ipsec: basic netns - tunnel โ httpd: mod_ssl smoke sanity โ tuned: tune-processes-through-perf โ trace: ftrace/tracer ๐ง โ CIFS Connectathon ๐ง โ POSIX pjd-fstest suites ๐ง โ jvm - DaCapo Benchmark Suite ๐ง โ jvm - jcstress tests ๐ง โ Memory function: kaslr ๐ง โ LTP: openposix test suite ๐ง โ Networking vnic: ipvlan/basic ๐ง โ audit: audit testsuite test ๐ง โ iotop: sanity ๐ง โ storage: dm/common
Host 2: โ Boot test โ selinux-policy: serge-testsuite ๐ง โ Storage blktests
x86_64: Host 1: โ Boot test โ Storage SAN device stress - megaraid_sas
Host 2: โ Boot test ๐ฅ xfstests - ext4 โกโกโก xfstests - xfs โกโกโก selinux-policy: serge-testsuite โกโกโก lvm thinp sanity โกโกโก storage: software RAID testing โกโกโก stress: stress-ng ๐ง โกโกโก IOMMU boot test ๐ง โกโกโก IPMI driver test ๐ง โกโกโก IPMItool loop stress test ๐ง โกโกโก power-management: cpupower/sanity test ๐ง โกโกโก Storage blktests
Host 3: โ Boot test โ Podman system integration test - as root โ Podman system integration test - as user โ LTP โ Loopdev Sanity โ Memory function: memfd_create โ AMTU (Abstract Machine Test Utility) โ Networking bridge: sanity โ Ethernet drivers sanity โ Networking MACsec: sanity โ Networking socket: fuzz โ Networking sctp-auth: sockopts test โ Networking: igmp conformance test โ Networking route: pmtu โ Networking route_func - local โ Networking route_func - forward โ Networking TCP: keepalive test โ Networking UDP: socket โ Networking tunnel: geneve basic test โ Networking tunnel: gre basic โ L2TP basic test โ Networking tunnel: vxlan basic โ Networking ipsec: basic netns - transport โ Networking ipsec: basic netns - tunnel โ httpd: mod_ssl smoke sanity โ tuned: tune-processes-through-perf โ pciutils: sanity smoke test โ ALSA PCM loopback test โ ALSA Control (mixer) Userspace Element test โ Usex - version 1.9-29 โ storage: SCSI VPD โ trace: ftrace/tracer ๐ง โ CIFS Connectathon ๐ง โ POSIX pjd-fstest suites ๐ง โ jvm - DaCapo Benchmark Suite ๐ง โ jvm - jcstress tests ๐ง โ Memory function: kaslr ๐ง โ LTP: openposix test suite ๐ง โ Networking vnic: ipvlan/basic ๐ง โ audit: audit testsuite test ๐ง โ iotop: sanity ๐ง โ storage: dm/common
Host 4: โ Boot test โ Storage SAN device stress - mpt3sas driver
Test sources: https://github.com/CKI-project/tests-beaker ๐ Pull requests are welcome for new tests or improvements to existing tests!
Aborted tests ------------- Tests that didn't complete running successfully are marked with โกโกโก. If this was caused by an infrastructure issue, we try to mark that explicitly in the report.
Waived tests ------------ If the test run included waived tests, they are marked with ๐ง. Such tests are executed but their results are not taken into account. Tests are waived when their results are not reliable enough, e.g. when they're just introduced or are being fixed.
Testing timeout --------------- We aim to provide a report within reasonable timeframe. Tests that haven't finished running yet are marked with โฑ.
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
greg k-h
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
[0]: While these are new ones and not matching what we saw previously (that's why they weren't filtered out), the following commits from block tree should likely fix them:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit... https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit... https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit... https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit...
Veronika
greg k-h
On Tue, Mar 24, 2020 at 07:12:07AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
Bisection to find the offending commit would be best.
thanks,
greg k-h
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:18:19 PM Subject: Re: ๐ฅ PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:12:07AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
Bisection to find the offending commit would be best.
This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!
Veronika
thanks,
greg k-h
On Tue, Mar 24, 2020 at 07:44:32AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:18:19 PM Subject: Re: ๐ฅ PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:12:07AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
Bisection to find the offending commit would be best.
This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!
I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.
And if you can't reproduce it, why would you expect me to know what is going on? :)
thanks,
greg k-h
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:47:27 PM Subject: Re: ๐ฅ PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:44:32AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:18:19 PM Subject: Re: ๐ฅ PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:12:07AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git Commit: 8b841eb697e1 - Linux 5.5.12-rc1
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: OK Tests: PANICKED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh...
One or more kernel tests failed:
ppc64le: ๐ฅ xfstests - ext4 aarch64: โ LTP x86_64: ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
Bisection to find the offending commit would be best.
This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!
I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.
I meant in general, as we do catch a bunch of such problems. Those usually require some input and testing from people who understand the subsystem to create a reliable reproducer.
For this particular failure, do you want to try out the patches I linked in the first email or should we waive them for now?
And if you can't reproduce it, why would you expect me to know what is going on? :)
Well you do know the kernel pretty well :) We also cc the test maintainers to help out if it's needed since they understand the failed area, though given the current situation around the world people's reactions might understandably be slower.
Veronika
thanks,
greg k-h
On Tue, Mar 24, 2020 at 08:42:35AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:47:27 PM Subject: Re: ๐ฅ PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:44:32AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "Veronika Kabatova" vkabatov@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "CKI Project" cki-project@redhat.com, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 12:18:19 PM Subject: Re: ๐ฅ PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 07:12:07AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Greg KH" greg@kroah.com To: "CKI Project" cki-project@redhat.com Cc: "Memory Management" mm-qe@redhat.com, "Ondrej Mosnacek" omosnace@redhat.com, "Linux Stable maillist" stable@vger.kernel.org, "Jan Stancek" jstancek@redhat.com, "LTP Mailing List" ltp@lists.linux.it Sent: Tuesday, March 24, 2020 7:22:13 AM Subject: Re: ๐ฅ PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)
On Tue, Mar 24, 2020 at 05:42:38AM -0000, CKI Project wrote: > > Hello, > > We ran automated tests on a recent commit from this kernel tree: > > Kernel repo: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git > Commit: 8b841eb697e1 - Linux 5.5.12-rc1 > > The results of these automated tests are provided below. > > Overall result: FAILED (see details below) > Merge: OK > Compile: OK > Tests: PANICKED > > All kernel binaries, config files, and logs are available for > download > here: > > https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawareh... > > One or more kernel tests failed: > > ppc64le: > ๐ฅ xfstests - ext4 > > aarch64: > โ LTP > > x86_64: > ๐ฅ xfstests - ext4
Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(
Remember, if something starts breaking, I need some way to find out what caused it to break...
Hi Greg,
do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?
Bisection to find the offending commit would be best.
This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!
I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.
I meant in general, as we do catch a bunch of such problems. Those usually require some input and testing from people who understand the subsystem to create a reliable reproducer.
For this particular failure, do you want to try out the patches I linked in the first email or should we waive them for now?
As this must be something new introduced by my current queue, why not find out what patch in there is causing the problem?
If this is caused by something NOT in my queue, and is also a problem upstream in Linus's tree, then you need to notify the upstream developers as there's nothing I can do about it, right?
If this is not upstream, but is a problem in the stable tree, then please submit the patches that fix the issue as documented in the stable kernel rules document for how to submit them.
To throw this horrible "something is BROKEN!" type email at me all the time just makes me want to delete it as I have no context at all.
thanks,
greg k-h
linux-stable-mirror@lists.linaro.org