Lkft-triage July 2025

lkft-triage@lists.linaro.org

2 participants
3 discussions

Daily cts/vts report for lkft - 2025-Jul-01

by lava＠validation.linaro.org

Total jobs: 216 Total errors: 37 (17.13%) LAVA errors: 0 (0.00%) Test errors: 26 (12.04%) Job errors: 6 (2.78%) Infra errors: 5 (2.31%) Canceled jobs: 0 (0.00%) Device type: qrb5165-rb5 Total jobs: 66 Total errors: 14 (21.21%) Error type: Test Error count: 9 (13.64%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 4 (6.06%) IDs: rb5-03: 8337984 8338247 8338258 rb5-07: 8338218 Error: No match for error type 'Test', message 'tradefed - adb device lost[4fc7b22]' Count: 1 (1.52%) IDs: rb5-06: 8338174 Error: Device NOT found! Count: 2 (3.03%) IDs: rb5-03: 8338156 rb5-07: 8338145 Error: No match for error type 'Test', message 'tradefed - adb device lost[93dcfd96]' Count: 1 (1.52%) IDs: rb5-01: 8338090 Error: No match for error type 'Test', message 'tradefed - adb device lost[74d67c95]' Count: 1 (1.52%) IDs: rb5-03: 8338060 Error type: Infrastructure Error count: 5 (7.58%) Error: Connection closed Count: 5 (7.58%) IDs: rb5-01: 8337985 8338029 8338032 rb5-05: 8338027 8338209 Device type: dragonboard-845c Total jobs: 124 Total errors: 19 (15.32%) Error type: Test Error count: 13 (10.48%) Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 598 seconds' Count: 9 (7.26%) IDs: db845c-01: 8337882 8337996 8338082 db845c-02: 8338266 db845c-03: 8338240 db845c-04: 8338053 db845c-06: 8338143 db845c-07: 8337961 8338278 Error: Device NOT found! Count: 1 (0.81%) IDs: db845c-02: 8338277 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 583 seconds' Count: 1 (0.81%) IDs: db845c-02: 8338013 Error: No match for error type 'Test', message 'lava-docker-test-shell timed out after 599 seconds' Count: 1 (0.81%) IDs: db845c-01: 8337992 Error: No match for error type 'Test', message 'tradefed - adb device lost[f3a58ae3]' Count: 1 (0.81%) IDs: db845c-04: 8337908 Error type: Job Error count: 6 (4.84%) Error: No match for error type 'Job', message 'login-action timed out after 873 seconds' Count: 4 (3.23%) IDs: db845c-01: 8338019 8338161 8338227 8338230 Error: No match for error type 'Job', message 'login-action timed out after 788 seconds' Count: 1 (0.81%) IDs: db845c-01: 8338057 Error: No match for error type 'Job', message 'login-action timed out after 872 seconds' Count: 1 (0.81%) IDs: db845c-01: 8338051 Device type: sm8550-hdk Total jobs: 20 Total errors: 1 (5.00%) Error type: Test Error count: 1 (5.00%) Error: Device NOT found! Count: 1 (5.00%) IDs: sm8550-hdk-01: 8338036 Device type: hi6220-hikey-r2 Total jobs: 3 Total errors: 3 (100.00%) Error type: Test Error count: 3 (100.00%) Error: No match for error type 'Test', message 'tradefed - adb device lost[191FC75C003312EE]' Count: 1 (33.33%) IDs: hikey-6220-r2-06: 8338188 Error: No match for error type 'Test', message 'The network seems not available, as the ping command failed' Count: 2 (66.67%) IDs: hikey-6220-r2-01: 8337888 hikey-6220-r2-05: 8338106 Device type: x86 Total jobs: 1 Total errors: 0 (0.00%) Device type: hi960-hikey Total jobs: 2 Total errors: 0 (0.00%)

4 hours, 44 minutes

Daily report for validation - 2025-Jul-01

by lava＠validation.linaro.org

Total jobs: 588 Total errors: 295 (50.17%) LAVA errors: 8 (1.36%) Test errors: 1 (0.17%) Job errors: 44 (7.48%) Infra errors: 242 (41.16%) Canceled jobs: 0 (0.00%) Device type: beaglebone-black Total jobs: 365 Total errors: 187 (51.23%) Error type: Infrastructure Error count: 175 (47.95%) Error: matched a bootloader error message Count: 6 (1.64%) IDs: beaglebone-black03: 4156508 4159191 4159195 4159231 4159308 4159460 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 41 seconds' Count: 84 (23.01%) IDs: beaglebone-black03: 4156155 4156213 4156262 4156331 4156421 4156453 4156859 4158750 4158754 4158757 4158771 4158775 4158779 4158787 4158791 4158797 4158806 4158810 4158816 4158820 4158824 4158833 4158836 4158842 4158848 4158851 4158854 4158863 4158864 4158871 4158875 4158905 4158913 4158928 4158941 4158949 4158952 4158963 4158968 4158972 4158974 4158978 4158984 4158987 4158995 4159002 4159013 4159016 4159045 4159049 4159056 4159060 4159064 4159073 4159080 4159093 4159097 4159100 4159104 4159108 4159149 4159159 4159189 4159192 4159211 4159215 4159228 4159230 4159232 4159237 4159333 4159336 4159353 4159357 4159366 4159372 4159390 4159402 4159410 4159414 4159418 4159435 4159437 4159441 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 39 seconds' Count: 20 (5.48%) IDs: beaglebone-black03: 4156209 4156476 4156804 4156814 4158783 4158813 4158916 4158920 4158961 4158966 4159041 4159089 4159176 4159181 4159225 4159350 4159381 4159423 4159426 4159433 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 40 seconds' Count: 33 (9.04%) IDs: beaglebone-black03: 4156174 4156327 4156338 4156342 4156374 4156425 4156733 4158765 4158839 4158845 4158884 4158898 4158924 4158933 4158937 4158945 4158955 4158982 4158989 4158997 4159003 4159030 4159033 4159052 4159077 4159126 4159132 4159147 4159155 4159200 4159219 4159370 4159429 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 42 seconds' Count: 9 (2.47%) IDs: beaglebone-black03: 4156436 4159036 4159068 4159102 4159169 4159204 4159222 4159387 4159406 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 38 seconds' Count: 16 (4.38%) IDs: beaglebone-black03: 4156335 4158858 4158867 4158902 4158909 4158970 4158976 4159014 4159128 4159145 4159165 4159172 4159208 4159235 4159383 4159398 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 44 seconds' Count: 6 (1.64%) IDs: beaglebone-black03: 4156619 4158793 4159084 4159106 4159161 4159394 Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 37 seconds' Count: 1 (0.27%) IDs: beaglebone-black03: 4159348 Error type: Bug Error count: 8 (2.19%) Error: No match for error type 'Bug', message 'Invalid max_end_time value passed to wait()' Count: 8 (2.19%) IDs: beaglebone-black03: 4156151 4156459 4156819 4158761 4158802 4158830 4158880 4159439 Error type: Job Error count: 4 (1.10%) Error: wait for prompt timed out Count: 4 (1.10%) IDs: beaglebone-black03: 4158879 4159185 4159346 4159375 Device type: bcm2837-rpi-3-b-32 Total jobs: 173 Total errors: 98 (56.65%) Error type: Job Error count: 31 (17.92%) Error: wait for prompt timed out Count: 31 (17.92%) IDs: rpi3-b-32-03: 4158808 4158811 4158818 4158868 4158873 4158907 4158922 4158943 4158947 4158957 4159008 4159028 4159043 4159062 4159071 4159078 4159095 4159134 4159157 4159170 4159183 4159190 4159194 4159213 4159359 4159369 4159412 4159419 4159421 4159424 4159430 Error type: Infrastructure Error count: 67 (38.73%) Error: No match for error type 'Infrastructure', message 'bootloader-commands timed out after 35 seconds' Count: 67 (38.73%) IDs: rpi3-b-32-03: 4158483 4158814 4158822 4158826 4158828 4158831 4158834 4158882 4158895 4158900 4158904 4158911 4158914 4158918 4158926 4158929 4158931 4158935 4158939 4158950 4158953 4158959 4159004 4159031 4159037 4159039 4159047 4159050 4159054 4159058 4159066 4159069 4159075 4159082 4159085 4159087 4159091 4159148 4159153 4159163 4159167 4159174 4159177 4159179 4159187 4159197 4159198 4159202 4159206 4159210 4159217 4159220 4159331 4159344 4159352 4159355 4159378 4159379 4159385 4159389 4159392 4159396 4159400 4159404 4159408 4159416 4159427 Device type: x15 Total jobs: 26 Total errors: 1 (3.85%) Error type: Job Error count: 1 (3.85%) Error: No match for error type 'Job', message 'login-action timed out after 274 seconds' Count: 1 (3.85%) IDs: x15-01: 4158876 Device type: dragonboard-410c Total jobs: 9 Total errors: 8 (88.89%) Error type: Job Error count: 8 (88.89%) Error: No match for error type 'Job', message 'Unable to update image rootfs: 'mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda, missing codepage or helper program, or other error.\n dmesg(1) may have more information after failed mount system call.'' Count: 8 (88.89%) IDs: dragonboard-410c-01: 4158888 4158891 4159020 4159022 4159122 4159140 4159328 4159362 Device type: dragonboard-845c Total jobs: 12 Total errors: 0 (0.00%) Device type: hi6220-hikey-r2 Total jobs: 2 Total errors: 1 (50.00%) Error type: Test Error count: 1 (50.00%) Error: lava-test-shell timed out Count: 1 (50.00%) IDs: hi6220-hikey-r2-02: 4158985 Device type: imx93-11x11-evk Total jobs: 1 Total errors: 0 (0.00%)

4 hours, 48 minutes

next-20250627: IOMMU DMA warning during NVMe I/O completion after 06cae0e3f61c

by Ben Copeland

A regression in the NVMe PCI driver triggers IOMMU DMA warnings during I/O completion on ARM64 systems. The issue manifests as drivers/iommu/dma-iommu.c warnings during the DMA unmapping process. IOMMU DMA unmapping warnings occur during NVMe completion processing. The warnings appear consistently during normal NVMe I/O operations. - CONFIG_ARM64_64K_PAGES=y Test Environment: * Ampere Altra - AmpereOne Regression Analysis: - New regression? Yes - Reproducibility? Yes First seen on next-20250627 Good: next-20250626 Bad: next-20250627 Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Test log [ 32.699872] WARNING: drivers/iommu/dma-iommu.c:1232 at iommu_dma_unmap_page+0xc4/0xe8, CPU#13: swapper/13/0 [ 32.714204] Modules linked in: cdc_ether usbnet sm3_ce nvme sha3_ce nvme_core xhci_pci_renesas arm_cspmu_module arm_spe_pmu ipmi_devintf arm_cmn ipmi_msghandler cppc_cpufreq fuse drm backlight ip_tables x_tables [ 32.732967] CPU: 13 UID: 0 PID: 0 Comm: swapper/13 Tainted: G W 6.16.0-rc3-next-20250627 #1 PREEMPT [ 32.743562] Tainted: [W]=WARN [ 32.749381] Hardware name: Inspur NF5280R7/Mitchell MB, BIOS 04.04.00004001 2025-02-04 22:23:30 02/04/2025 [ 32.759020] pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 32.768746] pc : iommu_dma_unmap_page+0xc4/0xe8 [ 32.776040] lr : iommu_dma_unmap_page+0x40/0xe8 [ 32.780559] sp : ffff8000801afde0 [ 32.783861] x29: ffff8000801afde0 x28: 0000000000000005 x27: fff00001d7d230f0 [ 32.790983] x26: 0000000000000000 x25: fff00001003da0c8 x24: 0000000000000002 [ 32.798106] x23: 0000000000000000 x22: 0000000000001000 x21: 00000000feed5000 [ 32.805229] x20: fff00001003da0c8 x19: fff00001d7d23000 x18: 0000000000080000 [ 32.812352] x17: 0000000000000040 x16: ffffae5c7594ea68 x15: 0000000000000000 [ 32.819474] x14: 000000000007ffff x13: 0000000000000001 x12: 000000000000002c [ 32.826597] x11: 00000000000fffff x10: ffffffffffffffff x9 : ffffae5c76358e60 [ 32.833719] x8 : ffff8000801afd68 x7 : ffffae5c76358a78 x6 : 00000000feed5001 [ 32.840842] x5 : 000000000000000d x4 : ffffae5c76358a78 x3 : 0000000000000000 [ 32.847965] x2 : 0000000000000000 x1 : fff00001027e54c0 x0 : 0000000000000000 [ 32.855088] Call trace: [ 32.857521] iommu_dma_unmap_page+0xc4/0xe8 (P) [ 32.862039] dma_unmap_page_attrs+0x238/0x2a0 [ 32.866385] nvme_unmap_data+0x1dc/0x278 [nvme] [ 32.870904] nvme_pci_complete_rq+0xb8/0x118 [nvme] [ 32.878632] blk_complete_reqs+0x5c/0x80 [ 32.885320] blk_done_softirq+0x28/0x40 [ 32.892006] handle_softirqs+0x16c/0x408 [ 32.896436] __do_softirq+0x1c/0x28 [ 32.899912] ____do_softirq+0x18/0x30 [ 32.903561] call_on_irq_stack+0x24/0x30 [ 32.907472] do_softirq_own_stack+0x24/0x38 [ 32.911642] __irq_exit_rcu+0xd4/0x118 [ 32.915378] irq_exit_rcu+0x18/0x48 [ 32.918854] el1_interrupt+0x38/0x68 [ 32.922418] el1h_64_irq_handler+0x18/0x28 [ 32.926502] el1h_64_irq+0x6c/0x70 [ 32.929891] cpuidle_enter_state+0xcc/0x4c0 (P) [ 32.934410] cpuidle_enter+0x40/0x60 [ 32.937972] do_idle+0x204/0x288 [ 32.941188] cpu_startup_entry+0x3c/0x50 [ 32.945098] secondary_start_kernel+0x140/0x168 [ 32.949617] __secondary_switched+0xc0/0xc8 [ 32.953788] ---[ end trace 0000000000000000 ]--- ## Bisection git bisect start # bad: [06cae0e3f61c4c1ef18726b817bbb88c29f81e57] nvme-pci: merge the simple PRP and SGL setup into a common helper git bisect bad 06cae0e3f61c4c1ef18726b817bbb88c29f81e57 # good: [d6c12c69ef4fa33e32ceda4a53991ace01401cd9] block: add scatterlist-less DMA mapping helpers git bisect good d6c12c69ef4fa33e32ceda4a53991ace01401cd9 # good: [07c81cbf438b769e0d673be3b5c021a424a4dc6f] nvme-pci: refactor nvme_pci_use_sgls git bisect good 07c81cbf438b769e0d673be3b5c021a424a4dc6f # first bad commit: [06cae0e3f61c4c1ef18726b817bbb88c29f81e57] nvme-pci: merge the simple PRP and SGL setup into a common helper ## Source * Kernel version: 6.16.0-rc3-next-20250627 * Git tree: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git * Git sha: 2aeda9592360c200085898a258c4754bfe879921 * Git describe: 6.16.0-rc3-next-20250627 * Project details: https://qa-reports.linaro.org/lkft/linux-next-master-ampere/build/next-2025… * Architectures: arm64 * Toolchains: gcc-13 * Kconfigs: gcc-13-lkftconfig-64k_page_size ## Build arm64 * Test LAVA log: https://lkft-staging.validation.linaro.org/scheduler/job/180925#L7760 * Build link: https://storage.tuxsuite.com/public/ampere/ci/builds/2z6SDAcW8aM7PBWZ8oJ82Q… * Kernel config: https://storage.tuxsuite.com/public/ampere/ci/builds/2z6SDAcW8aM7PBWZ8oJ82Q… ## Steps to reproduce: # https://storage.tuxsuite.com/public/ampere/ci/builds/2z6SDAcW8aM7PBWZ8oJ82Q… Linaro LKFT https://lkft.linaro.org

10 hours, 12 minutes

2025

2024

2023

2022

2021

2020

2019

2018

2017

Lkft-triage July 2025