arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/ Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
Tree: arm-soc Branch: local/for-next Git Describe: v4.4-rc3-483-gb48b5ceb7227 Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808 Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
arm:
multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y: bcm4708-smartrg-sr400ac: 1 failed lab rk3288-rock2-square: 1 failed lab
multi_v7_defconfig+CONFIG_LKDTM=y: at91-sama5d3_xplained: 1 failed lab bcm4708-smartrg-sr400ac: 1 failed lab exynos5422-odroidxu3: 1 failed lab rk3288-rock2-square: 1 failed lab
multi_v7_defconfig+CONFIG_PROVE_LOCKING=y: rk3288-rock2-square: 1 failed lab
multi_v7_defconfig+CONFIG_ARM_LPAE=y: rk3288-rock2-square: 1 failed lab
multi_v7_defconfig: bcm4708-smartrg-sr400ac: 1 failed lab rk3288-rock2-square: 1 failed lab rk3288-rock2-square_rootfs:nfs: 1 failed lab
arm64:
defconfig+CONFIG_LKDTM=y: juno: 1 failed lab
--- For more info write to info@kernelci.org
On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/ Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
Tree: arm-soc Branch: local/for-next Git Describe: v4.4-rc3-483-gb48b5ceb7227 Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808 Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
We got three regressions in the arm-soc for-next branch today, compared to yesterday:
* I did not merge any rockchip changes, but it looks like multi_v7_defconfig is now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the list below, but in the web interface). I put a diff of the defconfig file on http://pastebin.com/DHgzz1jr, but I don't anything that may have cause the regression.
* bcm4708-smartrg-sr400ac doesn't find its rootfs any more
[ 16.670994] No filesystem could mount root, tried: ext3 ext2 ext4 squashfs vfat msdos ntfs [ 16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
I merged some patches from Florian yesterday, but it could also be a problem with the defconfig changes
* Juno seems to be sick and doesn't like LAVA today:
Connected to serial03.lavalab. Escape character is '^]'. Command sent successfully. Command sent successfully. ErrorMessage: Failed to boot test image. Lava failed at action boot_linaro_image with error:Failed to boot test image. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/lava_dispatcher/job.py", line 381, in run action.run(**params) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/actions/boot_control.py", line 156, in run raise CriticalError("Failed to boot test image.") CriticalError: Failed to boot test image. Command sent successfully.
Arnd
arm:
multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y: bcm4708-smartrg-sr400ac: 1 failed lab rk3288-rock2-square: 1 failed lab multi_v7_defconfig+CONFIG_LKDTM=y: at91-sama5d3_xplained: 1 failed lab bcm4708-smartrg-sr400ac: 1 failed lab exynos5422-odroidxu3: 1 failed lab rk3288-rock2-square: 1 failed lab multi_v7_defconfig+CONFIG_PROVE_LOCKING=y: rk3288-rock2-square: 1 failed lab multi_v7_defconfig+CONFIG_ARM_LPAE=y: rk3288-rock2-square: 1 failed lab multi_v7_defconfig: bcm4708-smartrg-sr400ac: 1 failed lab rk3288-rock2-square: 1 failed lab rk3288-rock2-square_rootfs:nfs: 1 failed labarm64:
defconfig+CONFIG_LKDTM=y: juno: 1 failed lab
On Wed, 2015-12-16 at 11:17 +0100, Arnd Bergmann wrote:
On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/ Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
[...]
- Juno seems to be sick and doesn't like LAVA today:
Looks like a LAVA hickup, I downloaded the Image and btb and they appear to work fine on my board. Just to make sure I was running the right kernel...
# uname -a Linux genericarmv8 4.4.0-rc3+ #1 SMP PREEMPT Wed Dec 16 05:51:25 CET 2015 aarch64 GNU/Linux
Am Mittwoch, 16. Dezember 2015, 11:17:30 schrieb Arnd Bergmann:
On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb72 27/ Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
Tree: arm-soc Branch: local/for-next Git Describe: v4.4-rc3-483-gb48b5ceb7227 Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808 Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
We got three regressions in the arm-soc for-next branch today, compared to yesterday:
- I did not merge any rockchip changes, but it looks like multi_v7_defconfig
is now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the list below, but in the web interface). I put a diff of the defconfig file on http://pastebin.com/DHgzz1jr, but I don't anything that may have cause the regression.
I would also think more of a hickup of the kernelci systems. When looking at veyron-jerry it shows the tftp kernel transfer not completing (line "Waiting for the transfer"...) and then the board getting reset and booting the original chromeos after that.
On the rock2 there is also nothing standing out in terms of errors or so. So I guess it might be best to simply check if that is still present tomorrow.
Heiko
On 16 December 2015 at 02:17, Arnd Bergmann arnd@arndb.de wrote:
On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/ Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
Tree: arm-soc Branch: local/for-next Git Describe: v4.4-rc3-483-gb48b5ceb7227 Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808 Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
We got three regressions in the arm-soc for-next branch today, compared to yesterday:
I had the bot bisect these failures, results below.
- I did not merge any rockchip changes, but it looks like multi_v7_defconfig is now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the list below, but in the web interface). I put a diff of the defconfig file on http://pastebin.com/DHgzz1jr, but I don't anything that may have cause the regression.
The bisection pointed d2318f340253 ("arm64: dts: berlin4ct: add watchdog nodes") which is clearly a red herring. kernelci.org inserts the modules from the build into the ramdisk, and eudev will probe them. With the modules inserted into the ramdisk, it fails to reach a user space prompt. If I remove the modules from the ramdisk it boots fine[1]. I may manually bisect if we cannot spot the issue easily.
- bcm4708-smartrg-sr400ac doesn't find its rootfs any more
[ 16.670994] No filesystem could mount root, tried: ext3 ext2 ext4 squashfs vfat msdos ntfs [ 16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
I merged some patches from Florian yesterday, but it could also be a problem with the defconfig changes
The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig: Enable some drivers for LS1021A")
I've rebased my tree with arm-soc for-next, and reverted the commits listed above. Should know shortly if the reverts fix the boot issues.
- Juno seems to be sick and doesn't like LAVA today:
Connected to serial03.lavalab. Escape character is '^]'. Command sent successfully. Command sent successfully. ErrorMessage: Failed to boot test image. Lava failed at action boot_linaro_image with error:Failed to boot test image. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/lava_dispatcher/job.py", line 381, in run action.run(**params) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/actions/boot_control.py", line 156, in run raise CriticalError("Failed to boot test image.") CriticalError: Failed to boot test image. Command sent successfully.
Indeed, I'll look into this. Thanks for reporting.
Cheers,
Tyler
On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
- bcm4708-smartrg-sr400ac doesn't find its rootfs any more
[ 16.670994] No filesystem could mount root, tried: ext3 ext2 ext4 squashfs vfat msdos ntfs [ 16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
I merged some patches from Florian yesterday, but it could also be a problem with the defconfig changes
The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig: Enable some drivers for LS1021A")
I've rebased my tree with arm-soc for-next, and reverted the commits listed above. Should know shortly if the reverts fix the boot issues.
I've looked at the built-in drivers to see if they might do something on unrelated hardware (we've had bugs like that before), but didn't find anything: the watchdog and cpufreq drivers all look sane, everything else is a loadable module.
One possible explanation would be a buggy bootloader that fails to pass a correct initrd and/or dtb if the kernel image gets too big.
Do you know if there is an initrd that is passed on this platform?
Arnd
On 16 December 2015 at 12:16, Arnd Bergmann arnd@arndb.de wrote:
On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
- bcm4708-smartrg-sr400ac doesn't find its rootfs any more
[ 16.670994] No filesystem could mount root, tried: ext3 ext2 ext4 squashfs vfat msdos ntfs [ 16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
I merged some patches from Florian yesterday, but it could also be a problem with the defconfig changes
The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig: Enable some drivers for LS1021A")
I've rebased my tree with arm-soc for-next, and reverted the commits listed above. Should know shortly if the reverts fix the boot issues.
I've looked at the built-in drivers to see if they might do something on unrelated hardware (we've had bugs like that before), but didn't find anything: the watchdog and cpufreq drivers all look sane, everything else is a loadable module.
One possible explanation would be a buggy bootloader that fails to pass a correct initrd and/or dtb if the kernel image gets too big.
Do you know if there is an initrd that is passed on this platform?
There is, and if I disable only CONFIG_BLK_DEV_RAM the bcm4708-smartrg-sr400ac boots again. This platform runs a CFE bootloader, and this is these are the kernel arguments injected into the DT - "console=ttyS0,115200 initrd=0x4000000,8M root=/dev/ram0 ip=dhcp".
I'm still investigating the rk3288 failures, and will report back when I find something.
Cheers,
Tyler
On Wednesday 16 December 2015 13:51:10 Tyler Baker wrote:
On 16 December 2015 at 12:16, Arnd Bergmann arnd@arndb.de wrote:
On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
- bcm4708-smartrg-sr400ac doesn't find its rootfs any more
[ 16.670994] No filesystem could mount root, tried: ext3 ext2 ext4 squashfs vfat msdos ntfs [ 16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
I merged some patches from Florian yesterday, but it could also be a problem with the defconfig changes
The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig: Enable some drivers for LS1021A")
I've rebased my tree with arm-soc for-next, and reverted the commits listed above. Should know shortly if the reverts fix the boot issues.
I've looked at the built-in drivers to see if they might do something on unrelated hardware (we've had bugs like that before), but didn't find anything: the watchdog and cpufreq drivers all look sane, everything else is a loadable module.
One possible explanation would be a buggy bootloader that fails to pass a correct initrd and/or dtb if the kernel image gets too big.
Do you know if there is an initrd that is passed on this platform?
There is, and if I disable only CONFIG_BLK_DEV_RAM the bcm4708-smartrg-sr400ac boots again. This platform runs a CFE bootloader, and this is these are the kernel arguments injected into the DT - "console=ttyS0,115200 initrd=0x4000000,8M root=/dev/ram0 ip=dhcp".
Ok, so it's probably the initrd= argument that causes the problem: as long as CONFIG_BLK_DEV_INITRD was disabled, the code in arm_memblock_init would ignore the initrd that gets passed by the bootloader and just use the initramfs that is appended to the kernel.
There may also be some problem related to how the bootloader uses a command line rather than the appropriate ATAG to pass the initrd location.
Arnd
kernel-build-reports@lists.linaro.org
-
Arnd Bergmann -
Heiko Stübner -
Jon Medhurst (Tixy) -
kernelci.org bot -
Tyler Baker