Approximately 20% of devices are experiencing intermittent boot failures
with this kernel version. The issue appears to be related to auto login
failures, where an incorrect password is being detected on the serial
console during the login process.
This intermittent regression is noticed on stable-rc 6.15.5-rc2 and
Linux mainline master v6.16-rc5. This regressions is only specific
to the devices not on the qemu's.
Test environments:
- dragonboard-410c
- dragonboard-845c
- e850-96
- juno-r2
- rk3399-rock-pi-4b
- x86
Regression Analysis:
- New regression? Yes
- Reproducibility? 20% only
Test regression: 6.15.5-rc2 v6.16-rc5 auto login failed Login incorrect
Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org>
## log in problem
runner-ns46nmmj-project-40964107-concurrent-0 login: #
Password:
Login incorrect
runner-ns46nmmj-project-40964107-concurrent-0 login:
## Investigation
The following three patches were reverted and the system was re-tested.
The previously reported issues are no longer observed after applying the
reverts.
serial: imx: Restore original RXTL for console to fix data loss
commit f23c52aafb1675ab1d1f46914556d8e29cbbf7b3 upstream.
serial: core: restore of_node information in sysfs
commit d36f0e9a0002f04f4d6dd9be908d58fe5bd3a279 upstream.
tty: serial: uartlite: register uart driver in init
[ Upstream commit 6bd697b5fc39fd24e2aa418c7b7d14469f550a93 ]
Reference bug report lore link,
- https://lore.kernel.org/stable/CA+G9fYvidpyHTQ179dAJ4TSdhthC-Mtjuks5iQjMf+o…
## Test links
* log 1: https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.15.y/build/v6.15…
* log 2: https://qa-reports.linaro.org/api/testruns/29021720/log_file/
* log 3: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.16-rc5/te…
* Boot test: https://regressions.linaro.org/lkft/linux-stable-rc-linux-6.15.y/v6.15.4-26…
* LAVA jobs 1: https://lkft.validation.linaro.org/scheduler/job/8344153#L1186
* LAVA jobs 2: https://lkft.validation.linaro.org/scheduler/job/8343870#L1266
## Build
* kernel: 6.15.5-rc2
* git: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
* git commit: f6977c36decb0875e78bdb8599749bce1e84c753
* git describe: v6.15.4-264-gf6977c36decb
* test details:
https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.15.y/build/v6.15…
## Test Regressions (compared to v6.15.3-590-gd93bc5feded1)
* dragonboard-410c, boot
- clang-20-lkftconfig
- clang-nightly-lkftconfig-kselftest
- gcc-13-lkftconfig-debug
* dragonboard-845c, boot
- clang-20-lkftconfig
- korg-clang-20-lkftconfig-lto-thing
* dragonboard-845c-compat, boot
- gcc-13-lkftconfig-compat
* e850-96, boot
- gcc-13-lkftconfig-no-kselftest-frag
* juno-r2, boot
- clang-20-lkftconfig
- gcc-13-lkftconfig-debug
- gcc-13-lkftconfig-kselftest
* rk3399-rock-pi-4b, boot
- clang-20-lkftconfig
* x86, boot
- clang-20-lkftconfig
- clang-20-lkftconfig-no-kselftest-frag
- clang-nightly-lkftconfig-kselftest
- clang-nightly-lkftconfig-lto-thing
- gcc-13-defconfig-preempt_rt
- gcc-13-lkftconfig-no-kselftest-frag
--
Linaro LKFT
https://lkft.linaro.org
Total jobs: 151
Total errors: 39 (25.83%)
LAVA errors: 0 (0.00%)
Test errors: 29 (19.21%)
Job errors: 6 (3.97%)
Infra errors: 4 (2.65%)
Canceled jobs: 0 (0.00%)
Device type: hi6220-hikey-r2
Total jobs: 8
Total errors: 8 (100.00%)
Error type: Test
Error count: 7 (87.50%)
Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed'
Count: 6 (75.00%)
IDs:
hikey-6220-r2-05:
8347346 8350502 8350533 8350551
hikey-6220-r2-06:
8347429 8350467
Error: No match for error type 'Test', message 'tradefed - adb device lost[235D989C003B0752]'
Count: 1 (12.50%)
IDs:
hikey-6220-r2-03:
8347364
Error type: Infrastructure
Error count: 1 (12.50%)
Error: No match for error type 'Infrastructure', message 'wait-device-boardid timed out after 1173 seconds'
Count: 1 (12.50%)
IDs:
hikey-6220-r2-06:
8350515
Device type: sm8550-hdk
Total jobs: 17
Total errors: 0 (0.00%)
Device type: qrb5165-rb5
Total jobs: 36
Total errors: 7 (19.44%)
Error type: Test
Error count: 5 (13.89%)
Error: Device NOT found!
Count: 3 (8.33%)
IDs:
rb5-03:
8350356 8350406 8350453
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 595 seconds'
Count: 1 (2.78%)
IDs:
rb5-07:
8350365
Error: No match for error type 'Test', message 'tradefed - adb device lost[93dcfd96]'
Count: 1 (2.78%)
IDs:
rb5-01:
8347303
Error type: Infrastructure
Error count: 2 (5.56%)
Error: Connection closed
Count: 2 (5.56%)
IDs:
rb5-01:
8350425 8350447
Device type: dragonboard-845c
Total jobs: 86
Total errors: 23 (26.74%)
Error type: Job
Error count: 5 (5.81%)
Error: No match for error type 'Job', message 'job-cleanup timed out after 300 seconds'
Count: 1 (1.16%)
IDs:
db845c-08:
8347380
Error: No match for error type 'Job', message 'login-action timed out after 873 seconds'
Count: 2 (2.33%)
IDs:
db845c-01:
8346971 8347255
Error: No match for error type 'Job', message 'login-action timed out after 872 seconds'
Count: 2 (2.33%)
IDs:
db845c-01:
8346976 8346978
Error type: Test
Error count: 17 (19.77%)
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 596 seconds'
Count: 1 (1.16%)
IDs:
db845c-02:
8347379
Error: Device NOT found!
Count: 2 (2.33%)
IDs:
db845c-02:
8347250 8347259
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 298 seconds'
Count: 2 (2.33%)
IDs:
db845c-01:
8347251
db845c-03:
8346683
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds'
Count: 7 (8.14%)
IDs:
db845c-01:
8347206 8347236
db845c-02:
8346967 8347167 8347224
db845c-06:
8347249
db845c-07:
8346966
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 573 seconds'
Count: 1 (1.16%)
IDs:
db845c-07:
8347180
Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 53997 seconds'
Count: 1 (1.16%)
IDs:
db845c-08:
8346968
Error: No match for error type 'Test', message 'tradefed - adb device lost[476f370a]'
Count: 1 (1.16%)
IDs:
db845c-10:
8346965
Error: No match for error type 'Test', message 'tradefed - adb device lost[b6b742b9]'
Count: 1 (1.16%)
IDs:
db845c-01:
8346690
Error: No match for error type 'Test', message 'tradefed - adb device lost[d9b7d9b]'
Count: 1 (1.16%)
IDs:
db845c-05:
8346663
Error type: Infrastructure
Error count: 1 (1.16%)
Error: No match for error type 'Infrastructure', message 'Failed to clean after job'
Count: 1 (1.16%)
IDs:
db845c-03:
8346685
Device type: x86
Total jobs: 2
Total errors: 0 (0.00%)
Device type: dragonboard-410c
Total jobs: 1
Total errors: 1 (100.00%)
Error type: Job
Error count: 1 (100.00%)
Error: No match for error type 'Job', message 'login-action timed out after 1770 seconds'
Count: 1 (100.00%)
IDs:
laa-00001-dragonboard-410c:
8347975
Device type: hi960-hikey
Total jobs: 1
Total errors: 0 (0.00%)