Hi,
As reported before[1], we found another regression in 6.1 when doing performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT being enabled by default by the following upstream commit if you have the right config dependencies enabled (commit is introduced in v5.16-rc1):
""" commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 Author: Frederic Weisbecker frederic@kernel.org Date: Tue Sep 14 12:31:34 2021 +0200
sched: Provide Kconfig support for default dynamic preempt mode """
We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT disabled in different perf benchmarks (including UnixBench process creation and redis). The root cause is explained in the commit log below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
""" commit cc6003916ed46d7a67d91ee32de0f9138047d55f Author: Hyeonggon Yoo 42.hyeyoo@gmail.com Date: Sat Jan 21 12:39:42 2023 +0900
lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default
In workloads where this_cpu operations are frequently performed, enabling DEBUG_PREEMPT may result in significant increase in runtime overhead due to frequent invocation of __this_cpu_preempt_check() function.
This can be demonstrated through benchmarks such as hackbench where this configuration results in a 10% reduction in performance, primarily due to the added overhead within memcg charging path. """
[1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.c...
On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino luizcap@amazon.com wrote:
Hi,
As reported before[1], we found another regression in 6.1 when doing performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT being enabled by default by the following upstream commit if you have the right config dependencies enabled (commit is introduced in v5.16-rc1):
""" commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 Author: Frederic Weisbecker frederic@kernel.org Date: Tue Sep 14 12:31:34 2021 +0200
sched: Provide Kconfig support for default dynamic preempt mode """
We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT disabled in different perf benchmarks (including UnixBench process creation and redis). The root cause is explained in the commit log below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
Oh, I should've sent it to the stable. Thanks for sending it!
Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION) so distros that were using PREEMPT_VOLUNTARY are silently affected by that.
It looks appropriate to be backported to the stable tree (to me). Hmm but I think it should be backported to 5.15 too?
""" commit cc6003916ed46d7a67d91ee32de0f9138047d55f Author: Hyeonggon Yoo 42.hyeyoo@gmail.com Date: Sat Jan 21 12:39:42 2023 +0900
lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default In workloads where this_cpu operations are frequently performed, enabling DEBUG_PREEMPT may result in significant increase in runtime overhead due to frequent invocation of __this_cpu_preempt_check() function. This can be demonstrated through benchmarks such as hackbench where this configuration results in a 10% reduction in performance, primarily due to the added overhead within memcg charging path."""
[1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.c...
On Fri, Oct 20, 2023 at 11:46:36AM +0900, Hyeonggon Yoo wrote:
On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino luizcap@amazon.com wrote:
Hi,
As reported before[1], we found another regression in 6.1 when doing performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT being enabled by default by the following upstream commit if you have the right config dependencies enabled (commit is introduced in v5.16-rc1):
""" commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 Author: Frederic Weisbecker frederic@kernel.org Date: Tue Sep 14 12:31:34 2021 +0200
sched: Provide Kconfig support for default dynamic preempt mode """
We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT disabled in different perf benchmarks (including UnixBench process creation and redis). The root cause is explained in the commit log below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
Oh, I should've sent it to the stable. Thanks for sending it!
Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION) so distros that were using PREEMPT_VOLUNTARY are silently affected by that.
It looks appropriate to be backported to the stable tree (to me). Hmm but I think it should be backported to 5.15 too?
Now queued up, thanks.
greg k-h
On Fri, Oct 20, 2023 at 11:46:36AM +0900, Hyeonggon Yoo wrote:
On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino luizcap@amazon.com wrote:
Hi,
As reported before[1], we found another regression in 6.1 when doing performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT being enabled by default by the following upstream commit if you have the right config dependencies enabled (commit is introduced in v5.16-rc1):
""" commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 Author: Frederic Weisbecker frederic@kernel.org Date: Tue Sep 14 12:31:34 2021 +0200
sched: Provide Kconfig support for default dynamic preempt mode """
We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT disabled in different perf benchmarks (including UnixBench process creation and redis). The root cause is explained in the commit log below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
Oh, I should've sent it to the stable. Thanks for sending it!
Thanks for doing the original fix! :)
Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION) so distros that were using PREEMPT_VOLUNTARY are silently affected by that.
It looks appropriate to be backported to the stable tree (to me). Hmm but I think it should be backported to 5.15 too?
Yeah, I see that Greg applied it to 5.15 and 5.10 as well.
I posted it only for 6.1 because the worst case seems to happen after c597bfddc9 where CONFIG_DEBUG_PREEMPT may be enabled automatically. But having the fix in earlier kernels is certainly good as well.
- Luiz
""" commit cc6003916ed46d7a67d91ee32de0f9138047d55f Author: Hyeonggon Yoo 42.hyeyoo@gmail.com Date: Sat Jan 21 12:39:42 2023 +0900
lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default In workloads where this_cpu operations are frequently performed, enabling DEBUG_PREEMPT may result in significant increase in runtime overhead due to frequent invocation of __this_cpu_preempt_check() function. This can be demonstrated through benchmarks such as hackbench where this configuration results in a 10% reduction in performance, primarily due to the added overhead within memcg charging path."""
[1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.c...
linux-stable-mirror@lists.linaro.org
-
Greg KH -
Hyeonggon Yoo -
Luiz Capitulino