Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Regards,
Hans
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Regards,
Hans
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
Hi,
On 3/9/22 19:27, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
Ok, done.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
The first results from the Fedora test kernel builds are in:
"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart" https://bugzilla.redhat.com/show_bug.cgi?id=2059668
5.16.9: good 5.16.10+: bad 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 (plain): good
So this seems to match the arch-linux report and the email report you linked. There is a problem with the backport in 5.16.10+, while 5.17-rc7 is fine.
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits that would be.
Regards,
Hans
p.s.
I've also gotten results in for: "Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Appear from Kernel 5.16.10" but the results there are mixed. There might be some EC firmware issue in play there.
I'm getting the feeling that on the x1c9 there really are 2 issues which both only trigger sometimes, making testing this problematic. One of those 2 issues is likely the same 5.16.10+ issue reported on laptops from other vendors.
On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 19:27, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote:
Hi Rafael,
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
Ok, done.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
The first results from the Fedora test kernel builds are in:
"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart" https://bugzilla.redhat.com/show_bug.cgi?id=2059668
5.16.9: good 5.16.10+: bad 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 (plain): good
So this seems to match the arch-linux report and the email report you linked. There is a problem with the backport in 5.16.10+, while 5.17-rc7 is fine.
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits that would be.
I do have an idea, but regardless of this, IMO the least risky way forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" which has been backported, because it carried a Fixes tag and not because it was marked for "stable".
Let me do that.
Hi,
On 3/10/22 11:56, Rafael J. Wysocki wrote:
On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 19:27, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote: > > Hi Rafael, > > We (Fedora) have been receiving a whole bunch of bug reports about > laptops getting hot/toasty while suspended with kernels >= 5.16.10 > and this seems to still happen with 5.17-rc7 too. > > The following are all bugzilla.redhat.com bug numbers: > > 1750910 - Laptop failed to suspend and completely drained the battery > 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep > 2053957 - Package c-states never go below C2 > 2056729 - No lid events when closing lid / laptop does not suspend > 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap > 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge > 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 > > And one of the bugs has also been mirrored at bugzilla.kernel.org by > the reporter: > > bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 > > The common denominator here (besides the kernel version) seems to > be that these are all Ice or Tiger Lake systems (I did not do > check this applies 100% to all bugs, but it does see, to be a pattern). > > A similar arch-linux report: > > https://bbs.archlinux.org/viewtopic.php?id=274292&p=2 > > Suggest that reverting > "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" > > which was cherry-picked into 5.16.10 fixes things.
Thanks for letting me know!
> If you want I can create Fedora kernel test-rpms of a recent > 5.16.y with just that one commit reverted and ask users to > confirm if that helps. Please let me know if doing that woulkd > be useful ?
Yes, it would.
However, it follows from the arch-linux report linked above that 5.17-rc is fine, so it would be good to also check if reverting that commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
Ok, done.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
The first results from the Fedora test kernel builds are in:
"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart" https://bugzilla.redhat.com/show_bug.cgi?id=2059668
5.16.9: good 5.16.10+: bad 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 (plain): good
So this seems to match the arch-linux report and the email report you linked. There is a problem with the backport in 5.16.10+, while 5.17-rc7 is fine.
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits that would be.
I do have an idea, but regardless of this, IMO the least risky way forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" which has been backported, because it carried a Fixes tag and not because it was marked for "stable".
Let me do that.
Ok, that sounds good, thank you.
Regards,
Hans
+ KH
On 3/10/2022 06:22, Hans de Goede wrote:
Hi,
On 3/10/22 11:56, Rafael J. Wysocki wrote:
On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 19:27, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 14:57, Rafael J. Wysocki wrote: > On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote: >> >> Hi Rafael, >> >> We (Fedora) have been receiving a whole bunch of bug reports about >> laptops getting hot/toasty while suspended with kernels >= 5.16.10 >> and this seems to still happen with 5.17-rc7 too. >> >> The following are all bugzilla.redhat.com bug numbers: >> >> 1750910 - Laptop failed to suspend and completely drained the battery >> 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep >> 2053957 - Package c-states never go below C2 >> 2056729 - No lid events when closing lid / laptop does not suspend >> 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap >> 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge >> 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 >> >> And one of the bugs has also been mirrored at bugzilla.kernel.org by >> the reporter: >> >> bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 >> >> The common denominator here (besides the kernel version) seems to >> be that these are all Ice or Tiger Lake systems (I did not do >> check this applies 100% to all bugs, but it does see, to be a pattern). >> >> A similar arch-linux report: >> >> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2 >> >> Suggest that reverting >> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" >> >> which was cherry-picked into 5.16.10 fixes things. > > Thanks for letting me know! > >> If you want I can create Fedora kernel test-rpms of a recent >> 5.16.y with just that one commit reverted and ask users to >> confirm if that helps. Please let me know if doing that woulkd >> be useful ? > > Yes, it would. > > However, it follows from the arch-linux report linked above that > 5.17-rc is fine, so it would be good to also check if reverting that > commit from 5.17-rc helps.
Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
Ok, done.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
The first results from the Fedora test kernel builds are in:
"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart" https://bugzilla.redhat.com/show_bug.cgi?id=2059668
5.16.9: good 5.16.10+: bad 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 (plain): good
So this seems to match the arch-linux report and the email report you linked. There is a problem with the backport in 5.16.10+, while 5.17-rc7 is fine.
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits that would be.
I do have an idea, but regardless of this, IMO the least risky way forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" which has been backported, because it carried a Fixes tag and not because it was marked for "stable".
Let me do that.
Ok, that sounds good, thank you.
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure that it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other systems, I think we should consider doing that to solve it too.
On Mon, Mar 14, 2022 at 3:37 PM Limonciello, Mario mario.limonciello@amd.com wrote:
- KH
On 3/10/2022 06:22, Hans de Goede wrote:
Hi,
On 3/10/22 11:56, Rafael J. Wysocki wrote:
On Thu, Mar 10, 2022 at 10:07 AM Hans de Goede hdegoede@redhat.com wrote:
Hi,
On 3/9/22 19:27, Rafael J. Wysocki wrote:
On Wed, Mar 9, 2022 at 5:34 PM Rafael J. Wysocki rafael@kernel.org wrote:
On Wed, Mar 9, 2022 at 5:33 PM Hans de Goede hdegoede@redhat.com wrote: > > Hi, > > On 3/9/22 14:57, Rafael J. Wysocki wrote: >> On Wed, Mar 9, 2022 at 2:44 PM Hans de Goede hdegoede@redhat.com wrote: >>> >>> Hi Rafael, >>> >>> We (Fedora) have been receiving a whole bunch of bug reports about >>> laptops getting hot/toasty while suspended with kernels >= 5.16.10 >>> and this seems to still happen with 5.17-rc7 too. >>> >>> The following are all bugzilla.redhat.com bug numbers: >>> >>> 1750910 - Laptop failed to suspend and completely drained the battery >>> 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep >>> 2053957 - Package c-states never go below C2 >>> 2056729 - No lid events when closing lid / laptop does not suspend >>> 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap >>> 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge >>> 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 >>> >>> And one of the bugs has also been mirrored at bugzilla.kernel.org by >>> the reporter: >>> >>> bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10 >>> >>> The common denominator here (besides the kernel version) seems to >>> be that these are all Ice or Tiger Lake systems (I did not do >>> check this applies 100% to all bugs, but it does see, to be a pattern). >>> >>> A similar arch-linux report: >>> >>> https://bbs.archlinux.org/viewtopic.php?id=274292&p=2 >>> >>> Suggest that reverting >>> "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" >>> >>> which was cherry-picked into 5.16.10 fixes things. >> >> Thanks for letting me know! >> >>> If you want I can create Fedora kernel test-rpms of a recent >>> 5.16.y with just that one commit reverted and ask users to >>> confirm if that helps. Please let me know if doing that woulkd >>> be useful ? >> >> Yes, it would. >> >> However, it follows from the arch-linux report linked above that >> 5.17-rc is fine, so it would be good to also check if reverting that >> commit from 5.17-rc helps. > > Ok, I've done Fedora kernel builds of both 5.16.13 and 5.17-rc7 with > the patch reverted and asked the bug-reporters to test both.
Thanks!
Also, in the cases where people have not tested 5.17-rc7 without any reverts, it would be good to ask them to do so.
Ok, done.
I have received another report related to this issue where the problem is not present in 5.17-rc7 (see https://lore.kernel.org/linux-pm/CAJZ5v0hKXyTtb1Jk=wqNV9_mZKdf3mmwF4bPOcmADy...).
The first results from the Fedora test kernel builds are in:
"HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case gets very hot and requires a power button hold to restart" https://bugzilla.redhat.com/show_bug.cgi?id=2059668
5.16.9: good 5.16.10+: bad 5.16.13 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 with "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" reverted: good 5.17-rc7 (plain): good
So this seems to match the arch-linux report and the email report you linked. There is a problem with the backport in 5.16.10+, while 5.17-rc7 is fine.
It is likely that the commit in question actually depends on some other commits that were not backported into 5.16.y.
I was thinking the same thing, but I've no idea which commits that would be.
I do have an idea, but regardless of this, IMO the least risky way forward would be to request "stable" to drop "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE" which has been backported, because it carried a Fixes tag and not because it was marked for "stable".
Let me do that.
Ok, that sounds good, thank you.
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure that it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other systems, I think we should consider doing that to solve it too.
There is a series of ACPI EC driver commits that are present in 5.17-rc, but have not been included in any "stable" series:
befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop acpi_ec_delete_query() c33676aa4824 ACPI: EC: Make the event work state machine visible c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event() eafe7509ab8c ACPI: EC: Rename three functions a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler() 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler() 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler() 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query() ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()
It is likely that they prevent the problem exposed by the problematic commit from occurring, but I'm not sure which ones do that. Some of them are clearly cosmetic, but the ordering matters.
[Public]
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure that it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other systems, I think we should consider doing that to solve it too.
There is a series of ACPI EC driver commits that are present in 5.17-rc, but have not been included in any "stable" series:
befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop acpi_ec_delete_query() c33676aa4824 ACPI: EC: Make the event work state machine visible c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event() eafe7509ab8c ACPI: EC: Rename three functions a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler() 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler() 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler() 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query() ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()
It is likely that they prevent the problem exposed by the problematic commit from occurring, but I'm not sure which ones do that. Some of them are clearly cosmetic, but the ordering matters.
Hans,
Do you think you could get one of the folks who reported this regression to do a bisect to see which one "fixed" it? If we get lucky we can come down to some smaller hunks of code that can come back to stable instead of reverting.
On Wed, Mar 16, 2022 at 2:38 PM Limonciello, Mario Mario.Limonciello@amd.com wrote:
[Public]
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure that it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other systems, I think we should consider doing that to solve it too.
There is a series of ACPI EC driver commits that are present in 5.17-rc, but have not been included in any "stable" series:
befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop acpi_ec_delete_query() c33676aa4824 ACPI: EC: Make the event work state machine visible c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event() eafe7509ab8c ACPI: EC: Rename three functions a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler() 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler() 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler() 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query() ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()
It is likely that they prevent the problem exposed by the problematic commit from occurring, but I'm not sure which ones do that. Some of them are clearly cosmetic, but the ordering matters.
Hans,
Do you think you could get one of the folks who reported this regression to do a bisect to see which one "fixed" it? If we get lucky we can come down to some smaller hunks of code that can come back to stable instead of reverting.
It's been reverted already.
What we can do is to request adding it back along with other commits needed for it to work as expected.
Also, I think we'll need all of the commits listed up to and including c793570d8725 ("ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event()") at least, but that's just a guess.
[Public]
-----Original Message----- From: Rafael J. Wysocki rafael@kernel.org Sent: Wednesday, March 16, 2022 08:46 To: Limonciello, Mario Mario.Limonciello@amd.com Cc: Rafael J. Wysocki rafael@kernel.org; Hans de Goede hdegoede@redhat.com; Linux PM linux-pm@vger.kernel.org; Stable stable@vger.kernel.org; Justin Forbes jmforbes@linuxtx.org; Mark Pearson markpearson@lenovo.com; ACPI Devel Maling List <linux- acpi@vger.kernel.org>; Kai-Heng Feng kai.heng.feng@canonical.com Subject: Re: Many reports of laptops getting hot while suspended with kernels >= 5.16.10 || >= 5.17-rc1
On Wed, Mar 16, 2022 at 2:38 PM Limonciello, Mario Mario.Limonciello@amd.com wrote:
[Public]
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure
that
it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other
systems,
I think we should consider doing that to solve it too.
There is a series of ACPI EC driver commits that are present in 5.17-rc, but have not been included in any "stable" series:
befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop acpi_ec_delete_query() c33676aa4824 ACPI: EC: Make the event work state machine visible c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event() eafe7509ab8c ACPI: EC: Rename three functions a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler() 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler() 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler() 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query() ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()
It is likely that they prevent the problem exposed by the problematic commit from occurring, but I'm not sure which ones do that. Some of them are clearly cosmetic, but the ordering matters.
Hans,
Do you think you could get one of the folks who reported this regression to
do
a bisect to see which one "fixed" it? If we get lucky we can come down to some smaller hunks of code that can come back to stable instead of
reverting.
It's been reverted already.
What we can do is to request adding it back along with other commits needed for it to work as expected.
OK thanks, makes sense.
Also, I think we'll need all of the commits listed up to and including c793570d8725 ("ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event()") at least, but that's just a guess.
Yeah so we'll for sure need a bisect to confirm and come up with that list.
Hi,
On 3/16/22 14:37, Limonciello, Mario wrote:
[Public]
Just FWIW this fix that was backported to stable also fixed keyboard wakeup from s2idle on a number of HP laptops too. I know for sure that it fixed it on the AMD versions of them, and Kai Heng Feng suspected it will also fix it for the Intel versions. So if there is another commit that can be backported from 5.17 to make it safer for the other systems, I think we should consider doing that to solve it too.
There is a series of ACPI EC driver commits that are present in 5.17-rc, but have not been included in any "stable" series:
befd9b5b0c62 ACPI: EC: Relocate acpi_ec_create_query() and drop acpi_ec_delete_query() c33676aa4824 ACPI: EC: Make the event work state machine visible c793570d8725 ACPI: EC: Avoid queuing unnecessary work in acpi_ec_submit_event() eafe7509ab8c ACPI: EC: Rename three functions a105acd7e384 ACPI: EC: Simplify locking in acpi_ec_event_handler() 388fb77dcf97 ACPI: EC: Rearrange the loop in acpi_ec_event_handler() 98d364509d77 ACPI: EC: Fold acpi_ec_check_event() into acpi_ec_event_handler() 1f2350443dd2 ACPI: EC: Pass one argument to acpi_ec_query() ca8283dcd933 ACPI: EC: Call advance_transaction() from acpi_ec_dispatch_gpe()
It is likely that they prevent the problem exposed by the problematic commit from occurring, but I'm not sure which ones do that. Some of them are clearly cosmetic, but the ordering matters.
Hans,
Do you think you could get one of the folks who reported this regression to do a bisect to see which one "fixed" it?
We already know which commit is causing the regression. As Rafael already said the question is why things are not broken in 5.17 and that is not a straight forward bisect. So figuring this out is going to be a lot of work and I'm not sure of that it is worth it. I certainly don't have time to help users with debugging this.
If we get lucky we can come down to some smaller hunks of code that can come back to stable instead of reverting.
5.17 is almost done and in a couple of weeks Fedora (and Arch and other distros tracking the mainline kernel) will move to 5.17 resolving the wakeup by keyboard issue not working there.
5.16 is not a LTS kernel, so for other distros we would at a minimum figure out what needs to be backported to make things work with 5.15 making the delta / set of possible patches we need even bigger. So as already said IMHO this is not worth it, at least assuming that nothing bad happens when attempting wakeup by keyboard, iow it just does not work and does not put the laptop in some bad state?
Regards,
Hans
[Public]
Hans,
Do you think you could get one of the folks who reported this regression to
do
a bisect to see which one "fixed" it?
We already know which commit is causing the regression. As Rafael already said the question is why things are not broken in 5.17 and that is not a straight forward bisect. So figuring this out is going to be a lot of work and I'm not sure of that it is worth it. I certainly don't have time to help users with debugging this.
If we get lucky we can come down to some smaller hunks of code that can come back to stable instead of
reverting.
5.17 is almost done and in a couple of weeks Fedora (and Arch and other distros tracking the mainline kernel) will move to 5.17 resolving the wakeup by keyboard issue not working there.
5.16 is not a LTS kernel, so for other distros we would at a minimum figure out what needs to be backported to make things work with 5.15 making the delta / set of possible patches we need even bigger. So as already said IMHO this is not worth it, at least assuming that nothing bad happens when attempting wakeup by keyboard, iow it just does not work and does not put the laptop in some bad state?
Regards,
Hans
You're right, no bad state as a result and probably not worth the tradeoff for a bisect effort. OK, thanks.
Hi, this is your Linux kernel regression tracker.
On 09.03.22 14:44, Hans de Goede wrote:
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
I was about to sent a similar mail, but then I found this one. Thx for making my life easier. :-D
But could you do me a big favor and CC the regression mailing list (regressions@lists.linux.dev) in case similar situations arise in the future? tia!
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
Here is another, but it's basically linking to reports you already mentioned: https://bugzilla.kernel.org/show_bug.cgi?id=215661
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
From the thread I gather that it looks like 5.17 is not affected; if that changes, could anybody please give me a heads up please?
If you want I can create Fedora kernel test-rpms of a recent 5.16.y with just that one commit reverted and ask users to confirm if that helps. Please let me know if doing that woulkd be useful ?
FWIW: To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced 4287509b4d21e34dc49266c #regzbot ignore-activity
If it turns out this isn't a regression, free free to remove it from the tracking by sending a reply to this thread containing a paragraph like "#regzbot invalid: reason why this is invalid" (without the quotes).
Reminder for developers: when fixing the issue, please add a 'Link:' tags pointing to the report (the mail quoted above) using lore.kernel.org/r/, as explained in 'Documentation/process/submitting-patches.rst' and 'Documentation/process/5.Posting.rst'. Regzbot needs them to automatically connect reports with fixes, but they are useful in general, too.
I'm sending this to everyone that got the initial report, to make everyone aware of the tracking. I also hope that messages like this motivate people to directly get at least the regression mailing list and ideally even regzbot involved when dealing with regressions, as messages like this wouldn't be needed then. And don't worry, if I need to send other mails regarding this regression only relevant for regzbot I'll send them to the regressions lists only (with a tag in the subject so people can filter them away). With a bit of luck no such messages will be needed anyway.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I'm getting a lot of reports on my table. I can only look briefly into most of them and lack knowledge about most of the areas they concern. I thus unfortunately will sometimes get things wrong or miss something important. I hope that's not the case here; if you think it is, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
On Thu, Mar 10, 2022 at 11:02 AM Thorsten Leemhuis regressions@leemhuis.info wrote:
Hi, this is your Linux kernel regression tracker.
On 09.03.22 14:44, Hans de Goede wrote:
We (Fedora) have been receiving a whole bunch of bug reports about laptops getting hot/toasty while suspended with kernels >= 5.16.10 and this seems to still happen with 5.17-rc7 too.
I was about to sent a similar mail, but then I found this one. Thx for making my life easier. :-D
But could you do me a big favor and CC the regression mailing list (regressions@lists.linux.dev) in case similar situations arise in the future? tia!
The following are all bugzilla.redhat.com bug numbers:
1750910 - Laptop failed to suspend and completely drained the battery 2050036 - Framework laptop: 5.16.5 breaks s2idle sleep 2053957 - Package c-states never go below C2 2056729 - No lid events when closing lid / laptop does not suspend 2057909 - Thinkpad X1C 9th in s2idle suspend still draining battery to zero over night , Ap 2059668 - HP Envy Laptop deadlocks on entering suspend power state when plugged in. Case ge 2059688 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
And one of the bugs has also been mirrored at bugzilla.kernel.org by the reporter:
bko215641 - Dell G15 5510 s2idle fails in 5.16.11 works in 5.16.10
Here is another, but it's basically linking to reports you already mentioned: https://bugzilla.kernel.org/show_bug.cgi?id=215661
The common denominator here (besides the kernel version) seems to be that these are all Ice or Tiger Lake systems (I did not do check this applies 100% to all bugs, but it does see, to be a pattern).
A similar arch-linux report:
https://bbs.archlinux.org/viewtopic.php?id=274292&p=2
Suggest that reverting "ACPI: PM: s2idle: Cancel wakeup before dispatching EC GPE"
which was cherry-picked into 5.16.10 fixes things.
From the thread I gather that it looks like 5.17 is not affected; if
This is most likely correct.
There are at least 3 different sources that have confirmed that.
that changes, could anybody please give me a heads up please?
Sure.
linux-stable-mirror@lists.linaro.org
-
Hans de Goede -
Limonciello, Mario -
Limonciello, Mario
-
Rafael J. Wysocki -
Thorsten Leemhuis