A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de --- kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /* * Minimum number of threads to boot the kernel */ -#define MIN_THREADS 20 +#define MIN_THREADS 600
/* * Maximum number of threads
On Sat, Jul 12, 2025 at 01:08:29AM +0200, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /*
- Minimum number of threads to boot the kernel
*/ -#define MIN_THREADS 20 +#define MIN_THREADS 600 /*
- Maximum number of threads
-- 2.50.1
Hi,
This is the friendly patch-bot of Greg Kroah-Hartman. You have sent him a patch that has triggered this response. He used to manually respond to these common problems, but in order to save his sanity (he kept writing the same thing over and over, yet to different people), I was created. Hopefully you will not take offence and will fix the problem in your patch and resubmit it so that it can be accepted into the Linux kernel tree.
You are receiving this message because of the following common error(s) as indicated below:
- This looks like a new version of a previously submitted patch, but you did not list below the --- line any changes from the previous version. Please read the section entitled "The canonical patch format" in the kernel file, Documentation/process/submitting-patches.rst for what needs to be done here to properly describe this.
If you wish to discuss this problem further, or you have questions about how to resolve this issue, please feel free to respond to this email and Greg will reply once he has dug out from the pending patches received from other developers.
thanks,
greg k-h's patch email bot
Cc wqueue & umode helper folks
On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /*
- Minimum number of threads to boot the kernel
*/ -#define MIN_THREADS 20 +#define MIN_THREADS 600
As David noted, this is not the proper fix. It appears that usermode helper should use limited thread pool. I.e. instead of system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with max_active set to max_threads divided by some arbitrary constant (3? 4?)?
regards,
On Thu, 17 Jul 2025 07:26:59 +0200 Jiri Slaby jirislaby@kernel.org wrote:
Cc wqueue & umode helper folks
On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /*
- Minimum number of threads to boot the kernel
*/ -#define MIN_THREADS 20 +#define MIN_THREADS 600
As David noted, this is not the proper fix. It appears that usermode helper should use limited thread pool. I.e. instead of system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with max_active set to max_threads divided by some arbitrary constant (3? 4?)?
Or maybe just 1 ? I'd guess all the threads either block in the same place or just block each other??
David
regards,
On 7/17/25 23:34, David Laight wrote:
On Thu, 17 Jul 2025 07:26:59 +0200 Jiri Slaby jirislaby@kernel.org wrote:
Cc wqueue & umode helper folks
On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /* * Minimum number of threads to boot the kernel */ -#define MIN_THREADS 20 +#define MIN_THREADS 600
As David noted, this is not the proper fix. It appears that usermode helper should use limited thread pool. I.e. instead of system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with max_active set to max_threads divided by some arbitrary constant (3? 4?)?
Or maybe just 1 ? I'd guess all the threads either block in the same place or just block each other??
I will reduce the number of threads. Maybe to max 5 or maybe just one.
I think we should still increase the minimum number of threads, but something like 60 to 100 should be fine. It is calculated based RAM size 128MB RAM resulted already in 419 max threads.
Hauke
Hi,
I am not exactly sure how I should limit the number of parallel user mode helper calls. The user mode helper is calling wait_for_initramfs() so it could be that some calls are getting queued at the early bootup. This is probably the problem I am hitting.
I do not want to block the device creation till the user mode helper finished. This could also result in a deadlock and would probably slow down bootup.
When I limit the number of user mode helper calls to 1 and let the others wait in a system queue, I might block other unrelated tasks in the system queue.
I would create an own queue and let the async user mode helper wait in this queue to only execute one at a time. When one of them needs a long time in user space it would block the others. This workqueue would also be active all the time. After the bootup it would probably not do much work any more.
I do not like any of these solutions. Do you have better ideas?
Hauke
On 7/18/25 00:52, Hauke Mehrtens wrote:
On 7/17/25 23:34, David Laight wrote:
On Thu, 17 Jul 2025 07:26:59 +0200 Jiri Slaby jirislaby@kernel.org wrote:
Cc wqueue & umode helper folks
On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /* * Minimum number of threads to boot the kernel */ -#define MIN_THREADS 20 +#define MIN_THREADS 600
As David noted, this is not the proper fix. It appears that usermode helper should use limited thread pool. I.e. instead of system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with max_active set to max_threads divided by some arbitrary constant (3? 4?)?
Or maybe just 1 ? I'd guess all the threads either block in the same place or just block each other??
I will reduce the number of threads. Maybe to max 5 or maybe just one.
I think we should still increase the minimum number of threads, but something like 60 to 100 should be fine. It is calculated based RAM size 128MB RAM resulted already in 419 max threads.
Hauke
On Sun, 20 Jul 2025 17:28:20 +0200 Hauke Mehrtens hauke@hauke-m.de wrote:
Hi,
I am not exactly sure how I should limit the number of parallel user mode helper calls. The user mode helper is calling wait_for_initramfs() so it could be that some calls are getting queued at the early bootup. This is probably the problem I am hitting.
I do not want to block the device creation till the user mode helper finished. This could also result in a deadlock and would probably slow down bootup.
When I limit the number of user mode helper calls to 1 and let the others wait in a system queue, I might block other unrelated tasks in the system queue.
I would create an own queue and let the async user mode helper wait in this queue to only execute one at a time. When one of them needs a long time in user space it would block the others. This workqueue would also be active all the time. After the bootup it would probably not do much work any more.
I do not like any of these solutions. Do you have better ideas?
Could you put the requests onto a private queue but use a system work queue function the clear the queue - only starting the function if there isn't a copy running?
David
Hauke
On 7/18/25 00:52, Hauke Mehrtens wrote:
On 7/17/25 23:34, David Laight wrote:
On Thu, 17 Jul 2025 07:26:59 +0200 Jiri Slaby jirislaby@kernel.org wrote:
Cc wqueue & umode helper folks
On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
A modern Linux system creates much more than 20 threads at bootup. When I booted up OpenWrt in qemu the system sometimes failed to boot up when it wanted to create the 419th thread. The VM had 128MB RAM and the calculation in set_max_threads() calculated that max_threads should be set to 419. When the system booted up it tried to notify the user space about every device it created because CONFIG_UEVENT_HELPER was set and used. I counted 1299 calls to call_usermodehelper_setup(), all of them try to create a new thread and call the userspace hotplug script in it.
This fixes bootup of Linux on systems with low memory.
I saw the problem with qemu 10.0.2 using these commands: qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
Cc: stable@vger.kernel.org Signed-off-by: Hauke Mehrtens hauke@hauke-m.de
kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/fork.c b/kernel/fork.c index 7966c9a1c163..388299525f3c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -115,7 +115,7 @@ /* * Minimum number of threads to boot the kernel */ -#define MIN_THREADS 20 +#define MIN_THREADS 600
As David noted, this is not the proper fix. It appears that usermode helper should use limited thread pool. I.e. instead of system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with max_active set to max_threads divided by some arbitrary constant (3? 4?)?
Or maybe just 1 ? I'd guess all the threads either block in the same place or just block each other??
I will reduce the number of threads. Maybe to max 5 or maybe just one.
I think we should still increase the minimum number of threads, but something like 60 to 100 should be fine. It is calculated based RAM size 128MB RAM resulted already in 419 max threads.
Hauke
linux-stable-mirror@lists.linaro.org
-
David Laight -
Greg KH -
Hauke Mehrtens -
Jiri Slaby