On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
So we had a slight pickup in commits this last week, but as expected and hoped for, things were still pretty quiet. About twice as many commits as the holiday week, but that's still not all that many.
I expect things will start becoming more normal now that people are back from the holidays and are starting to recover and wake up from their food comas.
In the meantime, below is the shortlog for the last week. Nothing particularly stands out, the changes are dominated by various driver updates (gpu, rdma and networking), with a random smattering of fixes elsewhere.
Something not well baked managed to sneak in and it is tagged for stable:
adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Reverting it fixes the warn splat below.
[ 0.310373] smpboot: x86: Booting SMP configuration: [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 [ 0.313798] ------------[ cut here ]------------ [ 0.317530] workqueue: work disable count underflowed [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0 [ 0.317530] Modules linked in: [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11 [ 0.317530] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024 [ 0.317530] RIP: 0010:enable_work+0xa4/0xb0 [ 0.317530] Code: c0 48 83 c4 18 5b 5d e9 ca 25 9f 00 80 3d 0a c7 48 08 00 75 b3 c6 05 01 c7 48 08 01 90 48 c7 c7 c8 eb 1d 82 e8 5d 77 fd ff 90 <0f> 0b 90 90 eb 98 90 0f 0b 90 eb b1 90 90 90 90 90 90 90 90 90 90 [ 0.317530] RSP: 0018:ffffc90000137e18 EFLAGS: 00010082 [ 0.317530] RAX: 0000000000000029 RBX: ffff88807d66dda0 RCX: 00000000ffefffff [ 0.317530] RDX: 0000000000000001 RSI: ffffc90000137ce0 RDI: 0000000000000001 [ 0.317530] RBP: 0000000000000000 R08: 00000000ffefffff R09: 0000000000000058 [ 0.317530] R10: 0000000000000000 R11: ffffffff8244df00 R12: 0000000000000000 [ 0.317530] R13: ffff88807d6604e0 R14: ffffffff812439f0 R15: ffff88807d660508 [ 0.317530] FS: 0000000000000000(0000) GS:ffff88807d640000(0000) knlGS:0000000000000000 [ 0.317530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.317530] CR2: 0000000000000000 CR3: 0000000002c1a000 CR4: 00000000003506f0 [ 0.317530] Call Trace: [ 0.317530] <TASK> [ 0.317530] ? __warn+0xa0/0x160 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? report_bug+0x18c/0x1c0 [ 0.317530] ? handle_bug+0x54/0x90 [ 0.317530] ? exc_invalid_op+0x1b/0x80 [ 0.317530] ? asm_exc_invalid_op+0x1a/0x20 [ 0.317530] ? __pfx_vmstat_cpu_online+0x10/0x10 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? enable_work+0xa3/0xb0 [ 0.317530] vmstat_cpu_online+0x61/0x80 [ 0.317530] cpuhp_invoke_callback+0x10f/0x480 [ 0.317530] ? srso_return_thunk+0x5/0x5f [ 0.317530] cpuhp_thread_fun+0xd4/0x160 [ 0.317530] smpboot_thread_fn+0xdd/0x1f0 [ 0.317530] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 0.317530] kthread+0xca/0xf0 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork+0x50/0x60 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork_asm+0x1a/0x30 [ 0.317530] </TASK> [ 0.317530] ---[ end trace 0000000000000000 ]--- [ 0.377680] smp: Brought up 1 node, 16 CPUs [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
So we had a slight pickup in commits this last week, but as expected and hoped for, things were still pretty quiet. About twice as many commits as the holiday week, but that's still not all that many.
I expect things will start becoming more normal now that people are back from the holidays and are starting to recover and wake up from their food comas.
In the meantime, below is the shortlog for the last week. Nothing particularly stands out, the changes are dominated by various driver updates (gpu, rdma and networking), with a random smattering of fixes elsewhere.
Something not well baked managed to sneak in and it is tagged for stable:
adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Reverting it fixes the warn splat below.
[ 0.310373] smpboot: x86: Booting SMP configuration: [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 [ 0.313798] ------------[ cut here ]------------ [ 0.317530] workqueue: work disable count underflowed [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0 [ 0.317530] Modules linked in: [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11 [ 0.317530] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024 [ 0.317530] RIP: 0010:enable_work+0xa4/0xb0 [ 0.317530] Code: c0 48 83 c4 18 5b 5d e9 ca 25 9f 00 80 3d 0a c7 48 08 00 75 b3 c6 05 01 c7 48 08 01 90 48 c7 c7 c8 eb 1d 82 e8 5d 77 fd ff 90 <0f> 0b 90 90 eb 98 90 0f 0b 90 eb b1 90 90 90 90 90 90 90 90 90 90 [ 0.317530] RSP: 0018:ffffc90000137e18 EFLAGS: 00010082 [ 0.317530] RAX: 0000000000000029 RBX: ffff88807d66dda0 RCX: 00000000ffefffff [ 0.317530] RDX: 0000000000000001 RSI: ffffc90000137ce0 RDI: 0000000000000001 [ 0.317530] RBP: 0000000000000000 R08: 00000000ffefffff R09: 0000000000000058 [ 0.317530] R10: 0000000000000000 R11: ffffffff8244df00 R12: 0000000000000000 [ 0.317530] R13: ffff88807d6604e0 R14: ffffffff812439f0 R15: ffff88807d660508 [ 0.317530] FS: 0000000000000000(0000) GS:ffff88807d640000(0000) knlGS:0000000000000000 [ 0.317530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.317530] CR2: 0000000000000000 CR3: 0000000002c1a000 CR4: 00000000003506f0 [ 0.317530] Call Trace: [ 0.317530] <TASK> [ 0.317530] ? __warn+0xa0/0x160 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? report_bug+0x18c/0x1c0 [ 0.317530] ? handle_bug+0x54/0x90 [ 0.317530] ? exc_invalid_op+0x1b/0x80 [ 0.317530] ? asm_exc_invalid_op+0x1a/0x20 [ 0.317530] ? __pfx_vmstat_cpu_online+0x10/0x10 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? enable_work+0xa3/0xb0 [ 0.317530] vmstat_cpu_online+0x61/0x80 [ 0.317530] cpuhp_invoke_callback+0x10f/0x480 [ 0.317530] ? srso_return_thunk+0x5/0x5f [ 0.317530] cpuhp_thread_fun+0xd4/0x160 [ 0.317530] smpboot_thread_fn+0xdd/0x1f0 [ 0.317530] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 0.317530] kthread+0xca/0xf0 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork+0x50/0x60 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork_asm+0x1a/0x30 [ 0.317530] </TASK> [ 0.317530] ---[ end trace 0000000000000000 ]--- [ 0.377680] smp: Brought up 1 node, 16 CPUs [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
I've just dropped it from the stable queues as well.
thanks,
greg k-h
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
So we had a slight pickup in commits this last week, but as expected and hoped for, things were still pretty quiet. About twice as many commits as the holiday week, but that's still not all that many.
I expect things will start becoming more normal now that people are back from the holidays and are starting to recover and wake up from their food comas.
In the meantime, below is the shortlog for the last week. Nothing particularly stands out, the changes are dominated by various driver updates (gpu, rdma and networking), with a random smattering of fixes elsewhere.
Something not well baked managed to sneak in and it is tagged for stable:
adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Thanks for bissecting it.
I've just confirmed that reverting adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()") gets rid of the "workqueue: work disable count underflow" WARNING.
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
Hi All,
Something not well baked managed to sneak in and it is tagged for stable:
adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Reverting it fixes the warn splat below.
[ 0.310373] smpboot: x86: Booting SMP configuration: [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 [ 0.313798] ------------[ cut here ]------------ [ 0.317530] workqueue: work disable count underflowed [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0 [ 0.317530] Modules linked in: [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11
...
JFYI, hitting the same on s390:
[ 0.176304] smp: Bringing up secondary CPUs ... [ 0.186596] ------------[ cut here ]------------ [ 0.186603] workqueue: work disable count underflowed [ 0.186606] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0x10a/0x120 [ 0.186613] Modules linked in: [ 0.186615] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #206 [ 0.186619] Hardware name: IBM 3931 A01 701 (KVM/Linux) [ 0.186621] Krnl PSW : 0404c00180000000 00000308470979ee (enable_work+0x10e/0x120) [ 0.186626] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3 [ 0.186629] Krnl GPRS: 00000308484bf9b8 00000308484bf9b0 0000000000000029 00000308484bf9b8 [ 0.186632] 000002884717ba08 000002884717ba00 000003084726e5e0 000000000073e180 [ 0.186634] 000000007fffffff 0000000000000000 0000000000000000 000000007af24840 [ 0.186637] 0000000000e3a300 0000000000e3a300 00000308470979ea 000002884717bc60 [ 0.186685] Krnl Code: 00000308470979de: c02000846cd0 larl %r2,000003084812537e [ 0.186685] 00000308470979e4: c0e5fffee146 brasl %r14,0000030847073c70 [ 0.186685] #00000308470979ea: af000000 mc 0,0 [ 0.186685] >00000308470979ee: a7080000 lhi %r0,0 [ 0.186685] 00000308470979f2: a7f4ffb2 brc 15,0000030847097956 [ 0.186685] 00000308470979f6: 0707 bcr 0,%r7 [ 0.186685] 00000308470979f8: 0707 bcr 0,%r7 [ 0.186685] 00000308470979fa: 0707 bcr 0,%r7 [ 0.186708] Call Trace: [ 0.186710] [<00000308470979ee>] enable_work+0x10e/0x120 [ 0.186714] ([<00000308470979ea>] enable_work+0x10a/0x120) [ 0.186718] [<000003084726e64c>] vmstat_cpu_online+0x6c/0x90 [ 0.186722] [<0000030847075d9a>] cpuhp_invoke_callback+0x16a/0x4d0 [ 0.186725] [<0000030847076832>] cpuhp_thread_fun+0xc2/0x250 [ 0.186728] [<00000308470a8424>] smpboot_thread_fn+0xe4/0x1b0 [ 0.186731] [<00000308470a0d46>] kthread+0x126/0x130 [ 0.186735] [<000003084702508c>] __ret_from_fork+0x3c/0x60 [ 0.186738] [<0000030847d3a582>] ret_from_fork+0xa/0x38 [ 0.186742] Last Breaking-Event-Address: [ 0.186743] [<0000030847073d62>] __warn_printk+0xf2/0x100 [ 0.186746] Kernel panic - not syncing: kernel: panic_on_warn set ... [ 0.186749] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #206
Thanks!
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
So we had a slight pickup in commits this last week, but as expected and hoped for, things were still pretty quiet. About twice as many commits as the holiday week, but that's still not all that many.
I expect things will start becoming more normal now that people are back from the holidays and are starting to recover and wake up from their food comas.
In the meantime, below is the shortlog for the last week. Nothing particularly stands out, the changes are dominated by various driver updates (gpu, rdma and networking), with a random smattering of fixes elsewhere.
Something not well baked managed to sneak in and it is tagged for stable:
adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Reverting it fixes the warn splat below.
[ 0.310373] smpboot: x86: Booting SMP configuration: [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 [ 0.313798] ------------[ cut here ]------------ [ 0.317530] workqueue: work disable count underflowed [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0 [ 0.317530] Modules linked in: [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11 [ 0.317530] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024 [ 0.317530] RIP: 0010:enable_work+0xa4/0xb0 [ 0.317530] Code: c0 48 83 c4 18 5b 5d e9 ca 25 9f 00 80 3d 0a c7 48 08 00 75 b3 c6 05 01 c7 48 08 01 90 48 c7 c7 c8 eb 1d 82 e8 5d 77 fd ff 90 <0f> 0b 90 90 eb 98 90 0f 0b 90 eb b1 90 90 90 90 90 90 90 90 90 90 [ 0.317530] RSP: 0018:ffffc90000137e18 EFLAGS: 00010082 [ 0.317530] RAX: 0000000000000029 RBX: ffff88807d66dda0 RCX: 00000000ffefffff [ 0.317530] RDX: 0000000000000001 RSI: ffffc90000137ce0 RDI: 0000000000000001 [ 0.317530] RBP: 0000000000000000 R08: 00000000ffefffff R09: 0000000000000058 [ 0.317530] R10: 0000000000000000 R11: ffffffff8244df00 R12: 0000000000000000 [ 0.317530] R13: ffff88807d6604e0 R14: ffffffff812439f0 R15: ffff88807d660508 [ 0.317530] FS: 0000000000000000(0000) GS:ffff88807d640000(0000) knlGS:0000000000000000 [ 0.317530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.317530] CR2: 0000000000000000 CR3: 0000000002c1a000 CR4: 00000000003506f0 [ 0.317530] Call Trace: [ 0.317530] <TASK> [ 0.317530] ? __warn+0xa0/0x160 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? report_bug+0x18c/0x1c0 [ 0.317530] ? handle_bug+0x54/0x90 [ 0.317530] ? exc_invalid_op+0x1b/0x80 [ 0.317530] ? asm_exc_invalid_op+0x1a/0x20 [ 0.317530] ? __pfx_vmstat_cpu_online+0x10/0x10 [ 0.317530] ? enable_work+0xa4/0xb0 [ 0.317530] ? enable_work+0xa3/0xb0 [ 0.317530] vmstat_cpu_online+0x61/0x80 [ 0.317530] cpuhp_invoke_callback+0x10f/0x480 [ 0.317530] ? srso_return_thunk+0x5/0x5f [ 0.317530] cpuhp_thread_fun+0xd4/0x160 [ 0.317530] smpboot_thread_fn+0xdd/0x1f0 [ 0.317530] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 0.317530] kthread+0xca/0xf0 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork+0x50/0x60 [ 0.317530] ? __pfx_kthread+0x10/0x10 [ 0.317530] ret_from_fork_asm+0x1a/0x30 [ 0.317530] </TASK> [ 0.317530] ---[ end trace 0000000000000000 ]--- [ 0.377680] smp: Brought up 1 node, 16 CPUs [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
Thanks for letting me know, and apologies for the inconvenience caused. In the thread [1], I'm working on a follow-up fix with help from Lorenzo and others. Please feel free to take any necessary action (e.g. revert). Also, thank you, Greg, for [2].
[1]: https://lore.kernel.org/all/20241221033321.4154409-1-koichiro.den@canonical.... [2]: https://lore.kernel.org/stable/2025010603-tabasco-laziness-db0e@gregkh/
-Koichiro Den
-- Regards/Gruss, Boris.
On Mon, 6 Jan 2025 23:21:23 +0900 Koichiro Den koichiro.den@canonical.com wrote:
[ 0.377680] smp: Brought up 1 node, 16 CPUs [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
Thanks for letting me know, and apologies for the inconvenience caused. In the thread [1], I'm working on a follow-up fix with help from Lorenzo and others. Please feel free to take any necessary action (e.g. revert).
Thanks all, I have queued a revert.
On Mon, Jan 06, 2025 at 06:28:00PM -0800, Andrew Morton wrote:
Thanks all, I have queued a revert.
Linus zapped it already.
linux-stable-mirror@lists.linaro.org
-
Alexander Gordeev -
Andrew Morton -
Borislav Petkov -
Breno Leitao -
Greg KH -
Koichiro Den