Hello,
We are observing performance regressions (cpu usage, power consumption, dropped frames in video playback test, etc.) after updating to recent stable kernels. We tracked it down to commit 3cd2aa93674e in linux-6.1.y and commit 3cd2aa93674 in linux-6.6.y ("cpuidle: menu: Avoid discarding useful information", upstream commit 85975daeaa4).
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
On Tue, Oct 14, 2025 at 04:43:43PM +0900, Sergey Senozhatsky wrote:
Hello,
We are observing performance regressions (cpu usage, power consumption, dropped frames in video playback test, etc.) after updating to recent stable kernels. We tracked it down to commit 3cd2aa93674e in linux-6.1.y and commit 3cd2aa93674 in linux-6.6.y ("cpuidle: menu: Avoid discarding useful information", upstream commit 85975daeaa4).
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
For some reason that commit was not added to the 6.1 releases, sorry about that. Can you submit a working/tested backport so we can queue it up after the next round of releases in a few days?
thanks,
greg k-h
On (25/10/14 09:47), Greg Kroah-Hartman wrote:
On Tue, Oct 14, 2025 at 04:43:43PM +0900, Sergey Senozhatsky wrote:
Hello,
We are observing performance regressions (cpu usage, power consumption, dropped frames in video playback test, etc.) after updating to recent stable kernels. We tracked it down to commit 3cd2aa93674e in linux-6.1.y and commit 3cd2aa93674 in linux-6.6.y ("cpuidle: menu: Avoid discarding useful information", upstream commit 85975daeaa4).
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
For some reason that commit was not added to the 6.1 releases, sorry about that. Can you submit a working/tested backport so we can queue it up after the next round of releases in a few days?
Sorry for the confusion, the commit in question presents both in stable 6.1 and in 6.6 and appears to be causing regressions on our tests. I copy-pasted wrong commit id for 6.1: it should be a9edb700846 for 6.1 (and 3cd2aa93674 for 6.6).
On Tue, Oct 14, 2025 at 04:54:45PM +0900, Sergey Senozhatsky wrote:
On (25/10/14 09:47), Greg Kroah-Hartman wrote:
On Tue, Oct 14, 2025 at 04:43:43PM +0900, Sergey Senozhatsky wrote:
Hello,
We are observing performance regressions (cpu usage, power consumption, dropped frames in video playback test, etc.) after updating to recent stable kernels. We tracked it down to commit 3cd2aa93674e in linux-6.1.y and commit 3cd2aa93674 in linux-6.6.y ("cpuidle: menu: Avoid discarding useful information", upstream commit 85975daeaa4).
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
For some reason that commit was not added to the 6.1 releases, sorry about that. Can you submit a working/tested backport so we can queue it up after the next round of releases in a few days?
Sorry for the confusion, the commit in question presents both in stable 6.1 and in 6.6 and appears to be causing regressions on our tests. I copy-pasted wrong commit id for 6.1: it should be a9edb700846 for 6.1 (and 3cd2aa93674 for 6.6).
The point is still the same, commit fa3fa55de0d6 ("cpuidle: governors: menu: Avoid using invalid recent intervals data"), is not backported to 6.1.y, it is however in the following released kernels: 5.10.241 5.15.190 6.6.103 6.12.43 6.15.11 6.16.2 6.17 so something got lost in our trees and it needs to be backported.
I need to knock up a "what patches are missing" script again and sweep the trees to catch these types of things. It's been a year or so since I last did that.
thanks,
greg k-h
On (25/10/14 10:02), Greg Kroah-Hartman wrote:
The point is still the same, commit fa3fa55de0d6 ("cpuidle: governors: menu: Avoid using invalid recent intervals data"), is not backported to 6.1.y, it is however in the following released kernels: 5.10.241 5.15.190 6.6.103 6.12.43 6.15.11 6.16.2 6.17 so something got lost in our trees and it needs to be backported.
I can send a backport for 6.1, unless someone else wants to do it (or is already on it).
On 10/14/25 08:43, Sergey Senozhatsky wrote:
Hello,
We are observing performance regressions (cpu usage, power consumption, dropped frames in video playback test, etc.) after updating to recent stable kernels. We tracked it down to commit 3cd2aa93674e in linux-6.1.y and commit 3cd2aa93674 in linux-6.6.y ("cpuidle: menu: Avoid discarding useful information", upstream commit 85975daeaa4).
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
- 6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
- 6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
- 6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
Please find /sys/devices/system/cpu/cpu*/cpuidle/state*/* dumps attached (6 files).
The cpuidle/state*/* directory structure is as follows (NOTE: dumps don't contain s2idle subdirs):
/sys/devices/system/cpu/cpu1/cpuidle/ |-- state0 | |-- above | |-- below | |-- default_status | |-- desc | |-- disable | |-- latency | |-- name | |-- power | |-- rejected | |-- residency | |-- time | `-- usage |-- state1 | |-- above | |-- below | |-- default_status | |-- desc | |-- disable | |-- latency | |-- name | |-- power | |-- rejected | |-- residency | |-- s2idle | | |-- time | | `-- usage | |-- time | `-- usage |-- state2 | |-- above | |-- below | |-- default_status | |-- desc | |-- disable | |-- latency | |-- name | |-- power | |-- rejected | |-- residency | |-- s2idle | | |-- time | | `-- usage | |-- time | `-- usage `-- state3 |-- above |-- below |-- default_status |-- desc |-- disable |-- latency |-- name |-- power |-- rejected |-- residency |-- s2idle | |-- time | `-- usage |-- time `-- usage
Hope this helps.
On 10/14/25 12:55, Sergey Senozhatsky wrote:
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
This immediately threw me off. The fixup was written for a specific system which had completely broken cpuidle. It shouldn't affect any sane system significantly. I double checked the numbers and your system looks fine, in fact none of the tests had any rejected cpuidle occurrences. So functionally base and base-fixup are identical for you. The cpuidle numbers are also reasonably 'in the noise', so just for the future some stats would be helpful on those scores.
I can see a huge difference between base and revert in terms of cpuidle, so that's enough for me to take a look, I'll do that now. (6.1-revert has more C3_ACPI in favor of C1_ACPI.)
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
On Tue, Oct 14, 2025 at 5:11 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 12:55, Sergey Senozhatsky wrote:
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
This immediately threw me off. The fixup was written for a specific system which had completely broken cpuidle. It shouldn't affect any sane system significantly. I double checked the numbers and your system looks fine, in fact none of the tests had any rejected cpuidle occurrences. So functionally base and base-fixup are identical for you. The cpuidle numbers are also reasonably 'in the noise', so just for the future some stats would be helpful on those scores.
I can see a huge difference between base and revert in terms of cpuidle, so that's enough for me to take a look, I'll do that now. (6.1-revert has more C3_ACPI in favor of C1_ACPI.)
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
Well, yeah.
So I've already done some analysis.
There are 4 C-states, POLL, C1, C6 and C10 (at least that's what the MWAIT hints tell me).
This is how many times each of them was requested during the workload run on base 6.1.y:
POLL: 21445 C1: 2993722 C6: 767029 C10: 736854
and in percentage of the total idle state requests:
POLL: 0,47% C1: 66,25% C6: 16,97% C10: 16,31%
With the problematic commit reverted, this became
POLL: 16092 C1: 2452591 C6: 750933 C10: 1150259
and (again) in percentage of the total:
POLL: 0,37% C1: 56,12% C6: 17,18% C10: 26,32%
Overall, POLL is negligible and the revet had no effect on the number of times C6 was requested. The difference is for C1 and C10 and it's 10% in both cases, but going in opposite directions so to speak: C1 was requested 10% less and C10 was requested 10% more after the revert.
Let's see how this corresponds to the residency numbers.
For base 6.1.y there was
POLL: 599883 C1: 732303748 C6: 576785253 C10: 2020491489
and in percentage of the total
POLL: 0,02% C1: 21,99% C6: 17,32% C10: 60,67%
After the revert it became
POLL: 469451 C1: 517623465 C6: 508945687 C10: 2567701673
and in percentage of the total
POLL: 0,01% C1: 14,40% C6: 14,16% C10: 71,43%
so with the revert the CPUs spend around 7% more time in deep idle states (C6 and C10 combined).
I have to say that this is consistent with the intent of the problematic commit, which is to reduce the number of times the deepest idle state is requested although it is likely to be too deep.
However, on the system in question this somehow causes performance to drop significantly (even though shallow idle states are used more often which should result in lower average idle state exit latency and better performance).
One possible explanation is that this somehow affects turbo frequencies. That is, requesting shallower idle states on idle CPUs prevents the other CPUs from getting sufficiently high turbo.
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
On 10/14/25 16:54, Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 5:11 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 12:55, Sergey Senozhatsky wrote:
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote:
> Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using > invalid recent intervals data") doesn't address the problems we are > observing. Revert seems to be bringing performance metrics back to > pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
This immediately threw me off. The fixup was written for a specific system which had completely broken cpuidle. It shouldn't affect any sane system significantly. I double checked the numbers and your system looks fine, in fact none of the tests had any rejected cpuidle occurrences. So functionally base and base-fixup are identical for you. The cpuidle numbers are also reasonably 'in the noise', so just for the future some stats would be helpful on those scores.
I can see a huge difference between base and revert in terms of cpuidle, so that's enough for me to take a look, I'll do that now. (6.1-revert has more C3_ACPI in favor of C1_ACPI.)
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
Well, yeah.
So I've already done some analysis.
There are 4 C-states, POLL, C1, C6 and C10 (at least that's what the MWAIT hints tell me).
This is how many times each of them was requested during the workload run on base 6.1.y:
POLL: 21445 C1: 2993722 C6: 767029 C10: 736854
and in percentage of the total idle state requests:
POLL: 0,47% C1: 66,25% C6: 16,97% C10: 16,31%
With the problematic commit reverted, this became
POLL: 16092 C1: 2452591 C6: 750933 C10: 1150259
and (again) in percentage of the total:
POLL: 0,37% C1: 56,12% C6: 17,18% C10: 26,32%
Overall, POLL is negligible and the revet had no effect on the number of times C6 was requested. The difference is for C1 and C10 and it's 10% in both cases, but going in opposite directions so to speak: C1 was requested 10% less and C10 was requested 10% more after the revert.
Let's see how this corresponds to the residency numbers.
For base 6.1.y there was
POLL: 599883 C1: 732303748 C6: 576785253 C10: 2020491489
and in percentage of the total
POLL: 0,02% C1: 21,99% C6: 17,32% C10: 60,67%
After the revert it became
POLL: 469451 C1: 517623465 C6: 508945687 C10: 2567701673
and in percentage of the total
POLL: 0,01% C1: 14,40% C6: 14,16% C10: 71,43%
so with the revert the CPUs spend around 7% more time in deep idle states (C6 and C10 combined).
I have to say that this is consistent with the intent of the problematic commit, which is to reduce the number of times the deepest idle state is requested although it is likely to be too deep.
However, on the system in question this somehow causes performance to drop significantly (even though shallow idle states are used more often which should result in lower average idle state exit latency and better performance).
One possible explanation is that this somehow affects turbo frequencies. That is, requesting shallower idle states on idle CPUs prevents the other CPUs from getting sufficiently high turbo.
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
That's the most plausible explanation and would also be my guess. FWIW most of the C3_ACPI (== C10) with revert are objectively wrong with 78% idle misses (they were already pretty high with base around 72.5%).
I'll leave this here for easier following:
===== 6.1-base: after minus before deltas (aggregated across CPUs) ===== +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | state | time_diff_s | usage_diff | avg_resid_us | rejected_diff | above_diff | below_diff | share_% | +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | POLL | 0.600 | 21,445 | 28.0 | 0 | 0 | 19,846 | 0.02 | | C1_ACPI | 732.304 | 2,993,722 | 244.6 | 0 | 3,816 | 280,613 | 21.99 | | C2_ACPI | 576.785 | 767,029 | 752.0 | 0 | 272,105 | 453 | 17.32 | | C3_ACPI | 2,020.491 | 736,854 | 2,742.1 | 0 | 534,424 | 0 | 60.67 | | TOTAL | 3,330.180 | 4,519,050 | | 0 | 810,345 | 300,912 | 100.00 | +---------+-------------+------------+--------------+---------------+------------+------------+---------+
===== 6.1-revert: after minus before deltas (aggregated across CPUs) ===== +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | state | time_diff_s | usage_diff | avg_resid_us | rejected_diff | above_diff | below_diff | share_% | +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | POLL | 0.469 | 16,092 | 29.2 | 0 | 0 | 14,855 | 0.01 | | C1_ACPI | 517.623 | 2,452,591 | 211.1 | 0 | 4,109 | 150,500 | 14.40 | | C2_ACPI | 508.946 | 750,933 | 677.8 | 0 | 327,457 | 427 | 14.16 | | C3_ACPI | 2,567.702 | 1,150,259 | 2,232.3 | 0 | 895,311 | 0 | 71.43 | | TOTAL | 3,594.740 | 4,369,875 | | 0 | 1,226,877 | 165,782 | 100.00 | +---------+-------------+------------+--------------+---------------+------------+------------+---------+
===== 6.1-revert minus 6.1-base (state-by-state deltas of the deltas) ===== +---------+-----------+----------+----------+---------------+----------+----------+ | state | Δshare_pp | Δusage | Δtime_s | Δavg_resid_us | Δabove | Δbelow | +---------+-----------+----------+----------+---------------+----------+----------+ | POLL | -0.00 | -5,353 | -0.130 | 1.2 | +0 | -4,991 | | C1_ACPI | -7.59 | -541,131 | -214.680 | -33.6 | +293 | -130,113 | | C2_ACPI | -3.16 | -16,096 | -67.840 | -74.2 | +55,352 | -26 | | C3_ACPI | +10.76 | +413,405 | 547.210 | -509.8 | +360,887 | +0 | | TOTAL | +0.00 | -149,175 | 264.560 | | +416,532 | -135,130 | +---------+-----------+----------+----------+---------------+----------+----------+
On Tue, Oct 14, 2025 at 7:20 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 16:54, Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 5:11 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 12:55, Sergey Senozhatsky wrote:
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote:
On (25/10/14 10:50), Christian Loehle wrote: >> Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using >> invalid recent intervals data") doesn't address the problems we are >> observing. Revert seems to be bringing performance metrics back to >> pre-regression levels. > > Any details would be much appreciated. > How do the idle state usages differ with and without > "cpuidle: menu: Avoid discarding useful information"? > What do the idle states look like in your platform?
Sure, I can run tests. How do I get the numbers/stats that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
This immediately threw me off. The fixup was written for a specific system which had completely broken cpuidle. It shouldn't affect any sane system significantly. I double checked the numbers and your system looks fine, in fact none of the tests had any rejected cpuidle occurrences. So functionally base and base-fixup are identical for you. The cpuidle numbers are also reasonably 'in the noise', so just for the future some stats would be helpful on those scores.
I can see a huge difference between base and revert in terms of cpuidle, so that's enough for me to take a look, I'll do that now. (6.1-revert has more C3_ACPI in favor of C1_ACPI.)
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
Well, yeah.
So I've already done some analysis.
There are 4 C-states, POLL, C1, C6 and C10 (at least that's what the MWAIT hints tell me).
This is how many times each of them was requested during the workload run on base 6.1.y:
POLL: 21445 C1: 2993722 C6: 767029 C10: 736854
and in percentage of the total idle state requests:
POLL: 0,47% C1: 66,25% C6: 16,97% C10: 16,31%
With the problematic commit reverted, this became
POLL: 16092 C1: 2452591 C6: 750933 C10: 1150259
and (again) in percentage of the total:
POLL: 0,37% C1: 56,12% C6: 17,18% C10: 26,32%
Overall, POLL is negligible and the revet had no effect on the number of times C6 was requested. The difference is for C1 and C10 and it's 10% in both cases, but going in opposite directions so to speak: C1 was requested 10% less and C10 was requested 10% more after the revert.
Let's see how this corresponds to the residency numbers.
For base 6.1.y there was
POLL: 599883 C1: 732303748 C6: 576785253 C10: 2020491489
and in percentage of the total
POLL: 0,02% C1: 21,99% C6: 17,32% C10: 60,67%
After the revert it became
POLL: 469451 C1: 517623465 C6: 508945687 C10: 2567701673
and in percentage of the total
POLL: 0,01% C1: 14,40% C6: 14,16% C10: 71,43%
so with the revert the CPUs spend around 7% more time in deep idle states (C6 and C10 combined).
I have to say that this is consistent with the intent of the problematic commit, which is to reduce the number of times the deepest idle state is requested although it is likely to be too deep.
However, on the system in question this somehow causes performance to drop significantly (even though shallow idle states are used more often which should result in lower average idle state exit latency and better performance).
One possible explanation is that this somehow affects turbo frequencies. That is, requesting shallower idle states on idle CPUs prevents the other CPUs from getting sufficiently high turbo.
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
That's the most plausible explanation and would also be my guess. FWIW most of the C3_ACPI (== C10) with revert are objectively wrong with 78% idle misses (they were already pretty high with base around 72.5%).
Yeah, so the commit in question works as intended and from a purely cpuidle standpoint it is an improvement.
However, for the system/workload combination in question, cpuidle really is used as a means to crank up single-thread CPU performance.
Honestly, I'm not really sure what to do about this. It looks like this workload might benefit from selecting C6 more often, but I'm not quite sure how to make that happen in a generic way. Also, it may regress other workloads.
I'll leave this here for easier following:
Thanks!
===== 6.1-base: after minus before deltas (aggregated across CPUs) ===== +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | state | time_diff_s | usage_diff | avg_resid_us | rejected_diff | above_diff | below_diff | share_% | +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | POLL | 0.600 | 21,445 | 28.0 | 0 | 0 | 19,846 | 0.02 | | C1_ACPI | 732.304 | 2,993,722 | 244.6 | 0 | 3,816 | 280,613 | 21.99 | | C2_ACPI | 576.785 | 767,029 | 752.0 | 0 | 272,105 | 453 | 17.32 | | C3_ACPI | 2,020.491 | 736,854 | 2,742.1 | 0 | 534,424 | 0 | 60.67 | | TOTAL | 3,330.180 | 4,519,050 | | 0 | 810,345 | 300,912 | 100.00 | +---------+-------------+------------+--------------+---------------+------------+------------+---------+
===== 6.1-revert: after minus before deltas (aggregated across CPUs) ===== +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | state | time_diff_s | usage_diff | avg_resid_us | rejected_diff | above_diff | below_diff | share_% | +---------+-------------+------------+--------------+---------------+------------+------------+---------+ | POLL | 0.469 | 16,092 | 29.2 | 0 | 0 | 14,855 | 0.01 | | C1_ACPI | 517.623 | 2,452,591 | 211.1 | 0 | 4,109 | 150,500 | 14.40 | | C2_ACPI | 508.946 | 750,933 | 677.8 | 0 | 327,457 | 427 | 14.16 | | C3_ACPI | 2,567.702 | 1,150,259 | 2,232.3 | 0 | 895,311 | 0 | 71.43 | | TOTAL | 3,594.740 | 4,369,875 | | 0 | 1,226,877 | 165,782 | 100.00 | +---------+-------------+------------+--------------+---------------+------------+------------+---------+
===== 6.1-revert minus 6.1-base (state-by-state deltas of the deltas) ===== +---------+-----------+----------+----------+---------------+----------+----------+ | state | Δshare_pp | Δusage | Δtime_s | Δavg_resid_us | Δabove | Δbelow | +---------+-----------+----------+----------+---------------+----------+----------+ | POLL | -0.00 | -5,353 | -0.130 | 1.2 | +0 | -4,991 | | C1_ACPI | -7.59 | -541,131 | -214.680 | -33.6 | +293 | -130,113 | | C2_ACPI | -3.16 | -16,096 | -67.840 | -74.2 | +55,352 | -26 | | C3_ACPI | +10.76 | +413,405 | 547.210 | -509.8 | +360,887 | +0 | | TOTAL | +0.00 | -149,175 | 264.560 | | +416,532 | -135,130 | +---------+-----------+----------+----------+---------------+----------+----------+
On Tue, Oct 14, 2025 at 7:32 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Tue, Oct 14, 2025 at 7:20 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 16:54, Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 5:11 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 12:55, Sergey Senozhatsky wrote:
On (25/10/14 11:25), Christian Loehle wrote:
On 10/14/25 11:23, Sergey Senozhatsky wrote: > On (25/10/14 10:50), Christian Loehle wrote: >>> Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using >>> invalid recent intervals data") doesn't address the problems we are >>> observing. Revert seems to be bringing performance metrics back to >>> pre-regression levels. >> >> Any details would be much appreciated. >> How do the idle state usages differ with and without >> "cpuidle: menu: Avoid discarding useful information"? >> What do the idle states look like in your platform? > > Sure, I can run tests. How do I get the numbers/stats > that you are asking for?
Ideally just dump cat /sys/devices/system/cpu/cpu*/cpuidle/state*/* before and after the test.
OK, got some data for you. The terminology being used here is as follows:
6.1-base is 6.1 stable with a9edb700846 "cpuidle: menu: Avoid discarding useful information"
6.1-base-fixup is 6.1 stable with a9edb700846 and fa3fa55de0d6 "cpuidle: governors: menu: Avoid using invalid recent intervals data" cherry-pick
6.1-revert is 6.1 stable with a9edb700846 reverted (and no fixup commit, obviously)
Just to show the scale of regression, results of some of the benchmarks:
6.1-base: 84.5 6.1-base-fixup: 76.5 6.1-revert: 59.5
(lower is better, 6.1-revert has the same results as previous stable kernels).
This immediately threw me off. The fixup was written for a specific system which had completely broken cpuidle. It shouldn't affect any sane system significantly. I double checked the numbers and your system looks fine, in fact none of the tests had any rejected cpuidle occurrences. So functionally base and base-fixup are identical for you. The cpuidle numbers are also reasonably 'in the noise', so just for the future some stats would be helpful on those scores.
I can see a huge difference between base and revert in terms of cpuidle, so that's enough for me to take a look, I'll do that now. (6.1-revert has more C3_ACPI in favor of C1_ACPI.)
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
Well, yeah.
So I've already done some analysis.
There are 4 C-states, POLL, C1, C6 and C10 (at least that's what the MWAIT hints tell me).
This is how many times each of them was requested during the workload run on base 6.1.y:
POLL: 21445 C1: 2993722 C6: 767029 C10: 736854
and in percentage of the total idle state requests:
POLL: 0,47% C1: 66,25% C6: 16,97% C10: 16,31%
With the problematic commit reverted, this became
POLL: 16092 C1: 2452591 C6: 750933 C10: 1150259
and (again) in percentage of the total:
POLL: 0,37% C1: 56,12% C6: 17,18% C10: 26,32%
Overall, POLL is negligible and the revet had no effect on the number of times C6 was requested. The difference is for C1 and C10 and it's 10% in both cases, but going in opposite directions so to speak: C1 was requested 10% less and C10 was requested 10% more after the revert.
Let's see how this corresponds to the residency numbers.
For base 6.1.y there was
POLL: 599883 C1: 732303748 C6: 576785253 C10: 2020491489
and in percentage of the total
POLL: 0,02% C1: 21,99% C6: 17,32% C10: 60,67%
After the revert it became
POLL: 469451 C1: 517623465 C6: 508945687 C10: 2567701673
and in percentage of the total
POLL: 0,01% C1: 14,40% C6: 14,16% C10: 71,43%
so with the revert the CPUs spend around 7% more time in deep idle states (C6 and C10 combined).
I have to say that this is consistent with the intent of the problematic commit, which is to reduce the number of times the deepest idle state is requested although it is likely to be too deep.
However, on the system in question this somehow causes performance to drop significantly (even though shallow idle states are used more often which should result in lower average idle state exit latency and better performance).
One possible explanation is that this somehow affects turbo frequencies. That is, requesting shallower idle states on idle CPUs prevents the other CPUs from getting sufficiently high turbo.
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
That's the most plausible explanation and would also be my guess. FWIW most of the C3_ACPI (== C10) with revert are objectively wrong with 78% idle misses (they were already pretty high with base around 72.5%).
Yeah, so the commit in question works as intended and from a purely cpuidle standpoint it is an improvement.
However, for the system/workload combination in question, cpuidle really is used as a means to crank up single-thread CPU performance.
Honestly, I'm not really sure what to do about this. It looks like this workload might benefit from selecting C6 more often, but I'm not quite sure how to make that happen in a generic way. Also, it may regress other workloads.
Well, maybe the only thing that needs to be done is backporting
8de7606f0fe2 cpuidle: menu: Eliminate outliers on both ends of the sample set
(along with the dependencies) because if the workload tends to produce outliers on the low end of the idle duration data, it will effectively cause C1 to be selected more often after commit 85975daeaa4d ("cpuidle: menu: Avoid discarding useful information").
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
On 2025.10.14 18:30 Sergey Senozhatsky wrote:
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
The turbostat data suggests that power limit throttling is involved. It also suggests, but I am not sure, that temperature limiting measures might be involved.
We need to know more about the test system involved here. And we need to separate the variables. What thermal limiting methods are being used? Is idle injection being used? Or CPU frequency limiting or both. (I have very limited experience with thermald, and pretty much only use the TCC offset method.) Power and Thermal throttling is never involved when I test idle governor changes.
If it were me, I would limit the maximum CPU frequency such that power limit throttling did not engage for the test. That would likely also eliminate the need for any thermal limiting also. The suggestion is to then repeat the test.
From the discussion on this thread, it makes some sense that the selection of shallower idle states more often might have caused more throttling leading to the apparent benchmark regression.
... Doug
On 2025.10.14 21:50 Sergey Senozhatsky wrote:
On (25/10/14 20:41), Doug Smythies wrote:
What thermal limiting methods are being used? Is idle injection being used? Or CPU frequency limiting or both.
How do I find out?
From the turbostat data you do not appear to be using the TCC offset method. This line:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x0f690080 (105 C)
whereas on my test computer, using the TCC offset method, shows:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x14641422 (80 C) (100 default - 20 offset)
To check if thermal is being used do:
systemctl status thermal
Example:
doug@s19:~/idle/teo/menu2$ systemctl status thermald ○ thermald.service - Thermal Daemon Service Loaded: loaded (/usr/lib/systemd/system/thermald.service; disabled; preset: enabled) Active: inactive (dead)
If something else is being used, I don't know.
... Doug
On (25/10/15 07:11), Doug Smythies wrote:
What thermal limiting methods are being used? Is idle injection being used? Or CPU frequency limiting or both.
How do I find out?
From the turbostat data you do not appear to be using the TCC offset method. This line:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x0f690080 (105 C)
whereas on my test computer, using the TCC offset method, shows:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x14641422 (80 C) (100 default - 20 offset)
To check if thermal is being used do:
systemctl status thermal
chromeos doesn't use systemd. A quick ps grep doesn't show any thermal-related processes.
On Thu, Oct 16, 2025 at 6:59 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/15 07:11), Doug Smythies wrote:
What thermal limiting methods are being used? Is idle injection being used? Or CPU frequency limiting or both.
How do I find out?
From the turbostat data you do not appear to be using the TCC offset method. This line:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x0f690080 (105 C)
whereas on my test computer, using the TCC offset method, shows:
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x14641422 (80 C) (100 default - 20 offset)
To check if thermal is being used do:
systemctl status thermal
chromeos doesn't use systemd. A quick ps grep doesn't show any thermal-related processes.
All right, let's see what RAPL on that system has to say.
Please send the output of "grep . /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_*"
On (25/10/16 11:48), Rafael J. Wysocki wrote:
All right, let's see what RAPL on that system has to say.
Please send the output of "grep . /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_*"
/sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_max_power_uw:6000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_name:long_term /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw:6000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_time_window_us:27983872 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_max_power_uw:0 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_name:short_term /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw:12000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_time_window_us:976
On Thu, Oct 16, 2025 at 12:00 PM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/16 11:48), Rafael J. Wysocki wrote:
All right, let's see what RAPL on that system has to say.
Please send the output of "grep . /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_*"
/sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_max_power_uw:6000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_name:long_term /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw:6000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_time_window_us:27983872 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_max_power_uw:0 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_name:short_term /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw:12000000 /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_time_window_us:976
This looks reasonable, so I'd rather not recommend playing with it.
On Wed, Oct 15, 2025 at 5:41 AM Doug Smythies dsmythies@telus.net wrote:
On 2025.10.14 18:30 Sergey Senozhatsky wrote:
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
The turbostat data suggests that power limit throttling is involved.
Why do you think so?
On 2025.10.14 04:50 Rafael J. Wysocki wrote:
On Wed, Oct 15, 2025 at 5:41 AM Doug Smythies dsmythies@telus.net wrote:
On 2025.10.14 18:30 Sergey Senozhatsky wrote:
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
The turbostat data suggests that power limit throttling is involved.
Why do you think so?
I observed sustained processor package powers over the PL1 limit of 6 watts, combined with increased busy % while at the same time power and CPU frequency going down. I admit the time constant of 28 seconds only seems to be exceeded once.
Since I seem to be unable to function without making a graph, some are attached, including Idle state usage, which might be useful for your other email. (The graphs are poorly labelled and such.)
... Doug
On Wed, Oct 15, 2025 at 4:32 PM Doug Smythies dsmythies@telus.net wrote:
On 2025.10.14 04:50 Rafael J. Wysocki wrote:
On Wed, Oct 15, 2025 at 5:41 AM Doug Smythies dsmythies@telus.net wrote:
On 2025.10.14 18:30 Sergey Senozhatsky wrote:
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
The turbostat data suggests that power limit throttling is involved.
Why do you think so?
I observed sustained processor package powers over the PL1 limit of 6 watts, combined with increased busy % while at the same time power and CPU frequency going down. I admit the time constant of 28 seconds only seems to be exceeded once.
I see, thanks!
Since I seem to be unable to function without making a graph, some are attached, including Idle state usage, which might be useful for your other email.
Thanks for the graphs, they are useful.
(The graphs are poorly labelled and such.)
No worries.
On Wed, Oct 15, 2025 at 3:30 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/14 17:54), Rafael J. Wysocki wrote:
Sergey, can you please run the workload under turbostat on the base 6.1.y and on 6.1.y with the problematic commit reverted and send the turbostat output from both runs (note: turbostat needs to be run as root)?
Please find attached the turbostat logs for both cases.
Thanks!
First off, the CPUiD information reported by turbostat indicates that the system is a Jasper Lake. Is this correct?
Second, the overall picture I've got from the turbostat data indicates that there is a correlation between Bzy_MHz and CPU%c7 (the more time spent in C7, the higher the average "busy" frequency) which is generally consistent with the hypothesis that using more C1 causes the processor to drop the maximum frequency.
Something like this may be induced by RAPL power limits, presumably PL1.
Do you use thermald?
On (25/10/15 14:31), Rafael J. Wysocki wrote:
Please find attached the turbostat logs for both cases.
Thanks!
First off, the CPUiD information reported by turbostat indicates that the system is a Jasper Lake. Is this correct?
Correct. That particular board is powered by Jasper Lake.
[..]
Something like this may be induced by RAPL power limits, presumably PL1.
Do you use thermald?
I don't think so. chromeos doesn't use systemd.
On (25/10/14 16:11), Christian Loehle wrote:
(Also I can't send this email without at least recommending teo instead of menu for your platform / use-cases, if you deemed it unfit I'd love to know what didn't work for you!)
I can't tell for sure, I see a number of bugs evaluating teo vs menu, one of which mentions the following:
on Tigerlake/Alderlake hardware [..] TEO leads to higher power consumption (130mw more power usage in video conferencing) when compared to menu.
On Tue, Oct 14, 2025 at 12:23 PM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/14 10:50), Christian Loehle wrote:
Upstream fixup fa3fa55de0d ("cpuidle: governors: menu: Avoid using invalid recent intervals data") doesn't address the problems we are observing. Revert seems to be bringing performance metrics back to pre-regression levels.
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests.
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
On (25/10/14 15:47), Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 12:23 PM Sergey Senozhatsky senozhatsky@chromium.org wrote:
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests.
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
On 10/14/25 14:56, Sergey Senozhatsky wrote:
On (25/10/14 15:47), Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 12:23 PM Sergey Senozhatsky senozhatsky@chromium.org wrote:
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests.
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
On Tue, Oct 14, 2025 at 3:59 PM Christian Loehle christian.loehle@arm.com wrote:
On 10/14/25 14:56, Sergey Senozhatsky wrote:
On (25/10/14 15:47), Rafael J. Wysocki wrote:
On Tue, Oct 14, 2025 at 12:23 PM Sergey Senozhatsky senozhatsky@chromium.org wrote:
Any details would be much appreciated. How do the idle state usages differ with and without "cpuidle: menu: Avoid discarding useful information"? What do the idle states look like in your platform?
Sure, I can run tests.
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
Yeah, that'd be sufficiently close.
On (25/10/14 16:02), Rafael J. Wysocki wrote:
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
Yeah, that'd be sufficiently close.
Test results for menu.c from linux-next are within regressed range: 78.5
On Wed, Oct 15, 2025 at 3:56 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/14 16:02), Rafael J. Wysocki wrote:
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
Yeah, that'd be sufficiently close.
Test results for menu.c from linux-next are within regressed range: 78.5
So please check if the attached patch makes any difference.
On (25/10/15 15:08), Rafael J. Wysocki wrote:
On Wed, Oct 15, 2025 at 3:56 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/14 16:02), Rafael J. Wysocki wrote:
Would it be possible to check if the mainline has this issue? That is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
Yeah, that'd be sufficiently close.
Test results for menu.c from linux-next are within regressed range: 78.5
So please check if the attached patch makes any difference.
From what I can tell the patch fixes it!
On Thu, Oct 16, 2025 at 6:54 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/15 15:08), Rafael J. Wysocki wrote:
On Wed, Oct 15, 2025 at 3:56 AM Sergey Senozhatsky senozhatsky@chromium.org wrote:
On (25/10/14 16:02), Rafael J. Wysocki wrote:
> Would it be possible to check if the mainline has this issue? That > is, compare the benchmark results on unmodified 6.17 (say) and on 6.17 > with commit 85975daeaa4 reverted?
I don't think mainline kernel can run on those devices (due to a bunch of downstream patches). Best bet is 6.12, I guess.
Depending on what Rafael is expecting here you might just get away with copying menu.c from mainline, the interactions to other subsystems are limited fortunately.
Yeah, that'd be sufficiently close.
Test results for menu.c from linux-next are within regressed range: 78.5
So please check if the attached patch makes any difference.
From what I can tell the patch fixes it!
Well, that's something.
As far as I'm concerned, this change can be made.
From a purely theoretical standpoint, arguments can be made for doing it as well as for the current code and none of them is definitely preferable.
The current approach was chosen because it led to lower latency which should result in better performance, but after the patch the code is closer to what was done before.
Let me submit it officially and we'll see what people will have to say.
linux-stable-mirror@lists.linaro.org
-
Christian Loehle -
Doug Smythies -
Greg Kroah-Hartman -
Rafael J. Wysocki -
Sergey Senozhatsky