Hello everyone,
the systemd CI has [recently noticed][0] an issue within the ext4 file system after the Arch Linux kernel was upgraded to 6.16.1. The issue is exclusive to the stable tree and does not occur on 6.16 and not on 6.17-rc2. I have also tested 6.16.2-rc1 and it still contains the bug.
I was able to bisect the issue between 6.16 and 6.16.1 to the following commit:
b9c561f3f29c2 ("ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()")
The issue can be reproduced by running the tests from [TEST-58-REPART.sh][1] by running the [systemd integration tests][2]. But if there are any suggestions I can also test myself as the initial setup for the integration tests is a bit involved.
It is not yet clear to me whether this has real-world impact besides the test, but the systemd devs said that it's not a particularily demanding workflow, so I guess it is expected to work and could cause issues on other systems too.
Also does anybody have an idea which backport could be missing?
Cheers, Chris
[0]: https://github.com/systemd/systemd/actions/runs/17053272497/job/48345703316#... [1]: https://github.com/systemd/systemd/blob/main/test/units/TEST-58-REPART.sh [2]: https://github.com/systemd/systemd/blob/main/test/integration-tests/README.m...
---
#regzbot introduced: b9c561f3f29c2 #regzbot title: [STABLE] ext4: too many credits wanted / file system issue in v6.16.1 #regzbot link: https://github.com/systemd/systemd/actions/runs/17053272497/job/48345703316#...
---
git bisect start # status: waiting for both good and bad commits # good: [038d61fd642278bab63ee8ef722c50d10ab01e8f] Linux 6.16 git bisect good 038d61fd642278bab63ee8ef722c50d10ab01e8f # status: waiting for bad commit, 1 good commit known # bad: [3e0969c9a8c57ff3c6139c084673ebedfc1cf14f] Linux 6.16.1 git bisect bad 3e0969c9a8c57ff3c6139c084673ebedfc1cf14f # good: [288f1562e3f6af6d9b461eba49e75c84afa1b92c] media: v4l2-ctrls: Fix H264 SEPARATE_COLOUR_PLANE check git bisect good 288f1562e3f6af6d9b461eba49e75c84afa1b92c # bad: [f427460a1586c2e0865f9326b71ed6e5a0f404f2] f2fs: turn off one_time when forcibly set to foreground GC git bisect bad f427460a1586c2e0865f9326b71ed6e5a0f404f2 # bad: [5f57327f41a5bbb85ea382bc389126dd7b8f2d7b] scsi: elx: efct: Fix dma_unmap_sg() nents value git bisect bad 5f57327f41a5bbb85ea382bc389126dd7b8f2d7b # good: [9143c604415328d5dcd4d37b8adab8417afcdd21] leds: pca955x: Avoid potential overflow when filling default_label (take 2) git bisect good 9143c604415328d5dcd4d37b8adab8417afcdd21 # good: [9c4f20b7ac700e4b4377f85e36165a4f6ca85995] RDMA/hns: Fix accessing uninitialized resources git bisect good 9c4f20b7ac700e4b4377f85e36165a4f6ca85995 # good: [0b21d1962bec2e660c22c4c4231430f97163dcf8] perf tests bp_account: Fix leaked file descriptor git bisect good 0b21d1962bec2e660c22c4c4231430f97163dcf8 # good: [3dbe96d5481acd40d6090f174d2be8433d88716d] clk: thead: th1520-ap: Correctly refer the parent of osc_12m git bisect good 3dbe96d5481acd40d6090f174d2be8433d88716d # bad: [c6714f30ef88096a8da9fcafb6034dc4e9aa467d] clk: sunxi-ng: v3s: Fix de clock definition git bisect bad c6714f30ef88096a8da9fcafb6034dc4e9aa467d # bad: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks() git bisect bad b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8 # first bad commit: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
On Tue, 19 Aug 2025 at 21:53, Christian Heusel christian@heusel.eu wrote:
Hello everyone,
the systemd CI has [recently noticed][0] an issue within the ext4 file system after the Arch Linux kernel was upgraded to 6.16.1. The issue is exclusive to the stable tree and does not occur on 6.16 and not on 6.17-rc2. I have also tested 6.16.2-rc1 and it still contains the bug.
I was able to bisect the issue between 6.16 and 6.16.1 to the following commit:
b9c561f3f29c2 ("ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()")The issue can be reproduced by running the tests from [TEST-58-REPART.sh][1] by running the [systemd integration tests][2]. But if there are any suggestions I can also test myself as the initial setup for the integration tests is a bit involved.
It is not yet clear to me whether this has real-world impact besides the test, but the systemd devs said that it's not a particularily demanding workflow, so I guess it is expected to work and could cause issues on other systems too.
Also does anybody have an idea which backport could be missing?
Cheers, Chris
#regzbot introduced: b9c561f3f29c2 #regzbot title: [STABLE] ext4: too many credits wanted / file system issue in v6.16.1 #regzbot link: https://github.com/systemd/systemd/actions/runs/17053272497/job/48345703316#...
git bisect start # status: waiting for both good and bad commits # good: [038d61fd642278bab63ee8ef722c50d10ab01e8f] Linux 6.16 git bisect good 038d61fd642278bab63ee8ef722c50d10ab01e8f # status: waiting for bad commit, 1 good commit known # bad: [3e0969c9a8c57ff3c6139c084673ebedfc1cf14f] Linux 6.16.1 git bisect bad 3e0969c9a8c57ff3c6139c084673ebedfc1cf14f # good: [288f1562e3f6af6d9b461eba49e75c84afa1b92c] media: v4l2-ctrls: Fix H264 SEPARATE_COLOUR_PLANE check git bisect good 288f1562e3f6af6d9b461eba49e75c84afa1b92c # bad: [f427460a1586c2e0865f9326b71ed6e5a0f404f2] f2fs: turn off one_time when forcibly set to foreground GC git bisect bad f427460a1586c2e0865f9326b71ed6e5a0f404f2 # bad: [5f57327f41a5bbb85ea382bc389126dd7b8f2d7b] scsi: elx: efct: Fix dma_unmap_sg() nents value git bisect bad 5f57327f41a5bbb85ea382bc389126dd7b8f2d7b # good: [9143c604415328d5dcd4d37b8adab8417afcdd21] leds: pca955x: Avoid potential overflow when filling default_label (take 2) git bisect good 9143c604415328d5dcd4d37b8adab8417afcdd21 # good: [9c4f20b7ac700e4b4377f85e36165a4f6ca85995] RDMA/hns: Fix accessing uninitialized resources git bisect good 9c4f20b7ac700e4b4377f85e36165a4f6ca85995 # good: [0b21d1962bec2e660c22c4c4231430f97163dcf8] perf tests bp_account: Fix leaked file descriptor git bisect good 0b21d1962bec2e660c22c4c4231430f97163dcf8 # good: [3dbe96d5481acd40d6090f174d2be8433d88716d] clk: thead: th1520-ap: Correctly refer the parent of osc_12m git bisect good 3dbe96d5481acd40d6090f174d2be8433d88716d # bad: [c6714f30ef88096a8da9fcafb6034dc4e9aa467d] clk: sunxi-ng: v3s: Fix de clock definition git bisect bad c6714f30ef88096a8da9fcafb6034dc4e9aa467d # bad: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks() git bisect bad b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8 # first bad commit: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
The full kernel warning (immediately after the ext4 fs stops working):
[ 156.796652] JBD2: (sd-copy) wants too many credits credits:554 rsv_credits:0 max:338 [ 156.802468] ------------[ cut here ]------------ [ 156.802929] WARNING: CPU: 1 PID: 2904 at fs/jbd2/transaction.c:334 start_this_handle.cold+0x21/0x2e [ 156.803222] Modules linked in: dm_crypt encrypted_keys trusted asn1_encoder tee joydev mousedev iTCO_wdt intel_pmc_bxt iTCO_vendor_support kvm_amd i2c_i801 psmouse pcspkr vfat fat intel_agp mac_hid serio_raw lpc_ich intel_rapl_msr cfg80211 rfkill tun nfnetlink ip_tables intel_rapl_common ccp kvm irqbypass polyval_clmulni ghash_clmulni_intel sha512_ssse3 sha1_ssse3 aesni_intel virtio_balloon virtio_scsi i2c_smbus i2c_mux intel_gtt loop dm_mod qemu_fw_cfg vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock virtio_rng x_tables [ 156.804702] CPU: 1 UID: 0 PID: 2904 Comm: (sd-copy) Not tainted 6.16.1-arch1-1 #1 PREEMPT(full) 83823f140bb4fc8c507f38d1610ad9b642cd4b9a [ 156.805292] H[ 156.792161] TEST-70-TPM2.sh[2894]: Successfully forked off '(sd-copy)' as PID 2904.ardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022 [ 156.806972] RIP: 0010:start_this_handle.cold+0x21/0x2e [ 156.807210] Code: 4c 8b 14 24 e9 46 d7 64 00 65 48 8b 05 63 40 0d 03 44 89 f1 44 89 fa 48 c7 c7 30 84 3e a2 48 8d b0 f0 0c 00 00 e8 32 8b fe ff <0f> 0b 41 b9 e4 ff ff ff e9 dc d9 64 00 49 8b 54 24 18 49 8b 74 24 [ 156.807896] RSP: 0018:ffffd110c351f990 EFLAGS: 00010246 [ 156.808182] RAX: 0000000000000048 RBX: ffff8a8ed19b3000 RCX: 0000000000000000 [ 156.808367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8a8f3bd1cfc0 [ 156.808483] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000fffff35f [ 156.808591] R10: ffffffffa2e60de0 R11: ffffd110c351f830 R12: ffff8a8ec0775508 [ 156.808761] R13: 0000000000000005 R14: 0000000000000000 R15: 000000000000022a [ 156.808977] FS: 00007fa54aec3440(0000) GS:ffff8a8f9821b000(0000) knlGS:0000000000000000 [ 156.810623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 156.811062] CR2: 00005624ae24ce30 CR3: 00000001071d4002 CR4: 0000000000370ef0 [ 156.811288] Call Trace: [ 156.811402] <TASK> [ 156.811460] ? kmem_cache_alloc_noprof+0x12d/0x410 [ 156.811989] ? jbd2__journal_start+0x82/0x210 [ 156.812501] jbd2__journal_start+0xfc/0x210 [ 156.812507] ext4_truncate+0x168/0x440 [ 156.812512] ? unmap_mapping_range+0x80/0x130 [ 156.812517] ext4_setattr+0x706/0xa90 [ 156.812521] notify_change+0x340/0x4e0 [ 156.812526] ? do_truncate+0xc2/0xf0 [ 156.812529] do_truncate+0xc2/0xf0 [ 156.812533] [ 156.803411] TEST-70-TPM2.sh[2904]: Mounting /dev/mapper/luks-repart-2d18cd9c567b253b (ext4) on /run/systemd/mount-root (MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_NOATIME "")...do_ftruncate+0x12d/0x1f0 [ 156.812536] do_sys_ftruncate+0x40/0x80 [ 156.812539] __x64_sys_ftruncate+0x1a/0x30 [ 156.812541] do_syscall_64+0x81/0x970 [ 156.812546] ? shmem_file_llseek+0x6b/0xc0 [ 156.812551] ? syscall_exit_work+0x143/0x1b0 [ 156.812555] ? do_syscall_64+0x22f/0x970 [ 156.812557] ? syscall_exit_work+0x143/0x1b0 [ 156.812559] ? do_syscall_64+0x22f/0x970 [ 156.812563] ? syscall_exit_work+0x143/0x1b0 [ 156.812566] ? do_syscall_64+0x22f/0x970 [ 156.812568] ? shmem_file_llseek+0x6b/0xc0 [ 156.812571] ? syscall_exit_work+0x143/0x1b0 [ 156.812573] ? do_syscall_64+0x22f/0x970 [ 156.812575] ? do_syscall_64+0x22f/0x970 [ 156.812578] ? syscall_exit_work+0x143/0x1b0 [ 156.812580] ? do_syscall_64+0x22f/0x970 [ 156.812582] ? exc_page_fault+0x7e/0x1a0 [ 156.812585] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 156.812588] RIP: 0033:0x7fa54a110ceb [ 156.812595] Code: d2 31 f6 48 89 e5 48 83 ec 08 6a 4a e8 0e 25 f8 ff c9 c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 4d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 f1 7f 0f 00 f7 d8 [ 156.812597] RSP: 002b:00007ffd70e12d48 EFLAGS: 00000206 ORIG_RAX: 000000000000004d [ 156.812599] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007fa54a110ceb [ 156.812601] RDX: 0000000000000001 RSI: 0000000000000048 RDI: 0000000000000012 [ 156.812602] RBP: 00007ffd70e12e00 R08: 0000000000000000 R09: 00007fa54aec32d0 [ 156.812603] R10: 0000000000000008 R11: 0000000000000206 R12: 0000000000000013 [ 156.812604] R13: 7fffffffffffffb7 R14: 0000000000000048 R15: 0000000000010000 [ 156.812609] </TASK> [ 156.819965] ---[ end trace 0000000000000000 ]--- [ 156.819988] EXT4-fs error (device dm-0) in ext4_setattr:5987: error 28
On Tue, Aug 19, 2025 at 11:38:11PM +0100, Luca Boccassi wrote:
On Tue, 19 Aug 2025 at 21:53, Christian Heusel christian@heusel.eu wrote:
Hello everyone,
the systemd CI has [recently noticed][0] an issue within the ext4 file system after the Arch Linux kernel was upgraded to 6.16.1. The issue is exclusive to the stable tree and does not occur on 6.16 and not on 6.17-rc2. I have also tested 6.16.2-rc1 and it still contains the bug.
I was able to bisect the issue between 6.16 and 6.16.1 to the following commit:
b9c561f3f29c2 ("ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()")The issue can be reproduced by running the tests from [TEST-58-REPART.sh][1] by running the [systemd integration tests][2]. But if there are any suggestions I can also test myself as the initial setup for the integration tests is a bit involved.
It is not yet clear to me whether this has real-world impact besides the test, but the systemd devs said that it's not a particularily demanding workflow, so I guess it is expected to work and could cause issues on other systems too.
Also does anybody have an idea which backport could be missing?
Cheers, Chris
#regzbot introduced: b9c561f3f29c2 #regzbot title: [STABLE] ext4: too many credits wanted / file system issue in v6.16.1 #regzbot link: https://github.com/systemd/systemd/actions/runs/17053272497/job/48345703316#...
git bisect start # status: waiting for both good and bad commits # good: [038d61fd642278bab63ee8ef722c50d10ab01e8f] Linux 6.16 git bisect good 038d61fd642278bab63ee8ef722c50d10ab01e8f # status: waiting for bad commit, 1 good commit known # bad: [3e0969c9a8c57ff3c6139c084673ebedfc1cf14f] Linux 6.16.1 git bisect bad 3e0969c9a8c57ff3c6139c084673ebedfc1cf14f # good: [288f1562e3f6af6d9b461eba49e75c84afa1b92c] media: v4l2-ctrls: Fix H264 SEPARATE_COLOUR_PLANE check git bisect good 288f1562e3f6af6d9b461eba49e75c84afa1b92c # bad: [f427460a1586c2e0865f9326b71ed6e5a0f404f2] f2fs: turn off one_time when forcibly set to foreground GC git bisect bad f427460a1586c2e0865f9326b71ed6e5a0f404f2 # bad: [5f57327f41a5bbb85ea382bc389126dd7b8f2d7b] scsi: elx: efct: Fix dma_unmap_sg() nents value git bisect bad 5f57327f41a5bbb85ea382bc389126dd7b8f2d7b # good: [9143c604415328d5dcd4d37b8adab8417afcdd21] leds: pca955x: Avoid potential overflow when filling default_label (take 2) git bisect good 9143c604415328d5dcd4d37b8adab8417afcdd21 # good: [9c4f20b7ac700e4b4377f85e36165a4f6ca85995] RDMA/hns: Fix accessing uninitialized resources git bisect good 9c4f20b7ac700e4b4377f85e36165a4f6ca85995 # good: [0b21d1962bec2e660c22c4c4231430f97163dcf8] perf tests bp_account: Fix leaked file descriptor git bisect good 0b21d1962bec2e660c22c4c4231430f97163dcf8 # good: [3dbe96d5481acd40d6090f174d2be8433d88716d] clk: thead: th1520-ap: Correctly refer the parent of osc_12m git bisect good 3dbe96d5481acd40d6090f174d2be8433d88716d # bad: [c6714f30ef88096a8da9fcafb6034dc4e9aa467d] clk: sunxi-ng: v3s: Fix de clock definition git bisect bad c6714f30ef88096a8da9fcafb6034dc4e9aa467d # bad: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks() git bisect bad b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8 # first bad commit: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
The full kernel warning (immediately after the ext4 fs stops working):
I've pushed out a 6.16.3-rc1 that should hopefully resolve this.
thanks,
greg k-h
On 25/08/22 02:41PM, Greg KH wrote:
On Tue, Aug 19, 2025 at 11:38:11PM +0100, Luca Boccassi wrote:
On Tue, 19 Aug 2025 at 21:53, Christian Heusel christian@heusel.eu wrote:
Hello everyone,
the systemd CI has [recently noticed][0] an issue within the ext4 file system after the Arch Linux kernel was upgraded to 6.16.1. The issue is exclusive to the stable tree and does not occur on 6.16 and not on 6.17-rc2. I have also tested 6.16.2-rc1 and it still contains the bug.
I was able to bisect the issue between 6.16 and 6.16.1 to the following commit:
b9c561f3f29c2 ("ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()")The issue can be reproduced by running the tests from [TEST-58-REPART.sh][1] by running the [systemd integration tests][2]. But if there are any suggestions I can also test myself as the initial setup for the integration tests is a bit involved.
It is not yet clear to me whether this has real-world impact besides the test, but the systemd devs said that it's not a particularily demanding workflow, so I guess it is expected to work and could cause issues on other systems too.
Also does anybody have an idea which backport could be missing?
Cheers, Chris
#regzbot introduced: b9c561f3f29c2 #regzbot title: [STABLE] ext4: too many credits wanted / file system issue in v6.16.1 #regzbot link: https://github.com/systemd/systemd/actions/runs/17053272497/job/48345703316#...
git bisect start # status: waiting for both good and bad commits # good: [038d61fd642278bab63ee8ef722c50d10ab01e8f] Linux 6.16 git bisect good 038d61fd642278bab63ee8ef722c50d10ab01e8f # status: waiting for bad commit, 1 good commit known # bad: [3e0969c9a8c57ff3c6139c084673ebedfc1cf14f] Linux 6.16.1 git bisect bad 3e0969c9a8c57ff3c6139c084673ebedfc1cf14f # good: [288f1562e3f6af6d9b461eba49e75c84afa1b92c] media: v4l2-ctrls: Fix H264 SEPARATE_COLOUR_PLANE check git bisect good 288f1562e3f6af6d9b461eba49e75c84afa1b92c # bad: [f427460a1586c2e0865f9326b71ed6e5a0f404f2] f2fs: turn off one_time when forcibly set to foreground GC git bisect bad f427460a1586c2e0865f9326b71ed6e5a0f404f2 # bad: [5f57327f41a5bbb85ea382bc389126dd7b8f2d7b] scsi: elx: efct: Fix dma_unmap_sg() nents value git bisect bad 5f57327f41a5bbb85ea382bc389126dd7b8f2d7b # good: [9143c604415328d5dcd4d37b8adab8417afcdd21] leds: pca955x: Avoid potential overflow when filling default_label (take 2) git bisect good 9143c604415328d5dcd4d37b8adab8417afcdd21 # good: [9c4f20b7ac700e4b4377f85e36165a4f6ca85995] RDMA/hns: Fix accessing uninitialized resources git bisect good 9c4f20b7ac700e4b4377f85e36165a4f6ca85995 # good: [0b21d1962bec2e660c22c4c4231430f97163dcf8] perf tests bp_account: Fix leaked file descriptor git bisect good 0b21d1962bec2e660c22c4c4231430f97163dcf8 # good: [3dbe96d5481acd40d6090f174d2be8433d88716d] clk: thead: th1520-ap: Correctly refer the parent of osc_12m git bisect good 3dbe96d5481acd40d6090f174d2be8433d88716d # bad: [c6714f30ef88096a8da9fcafb6034dc4e9aa467d] clk: sunxi-ng: v3s: Fix de clock definition git bisect bad c6714f30ef88096a8da9fcafb6034dc4e9aa467d # bad: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks() git bisect bad b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8 # first bad commit: [b9c561f3f29c2d6e1c1d3ffc202910bef250b7d8] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
The full kernel warning (immediately after the ext4 fs stops working):
I've pushed out a 6.16.3-rc1 that should hopefully resolve this.
thanks,
greg k-h
Hey Greg,
6.16.3 does indeed resolve the issue, thanks for fixing :)
Cheers, Chris
linux-stable-mirror@lists.linaro.org
-
Christian Heusel -
Greg KH -
Luca Boccassi