Dear developers,
(sorry for the long CC list, it looks quite long to me, but I tried to follow the issue reporting guide as closely as possible)
Since patches [1], [2] and [3] were applied to the kernel, there is a regression with Lenovo ThinkPad Compact USB Keyboard (old model, not II).
[1] https://github.com/torvalds/linux/commit/46a0a2c96f0f47628190f122c2e3d879e59... [2] https://github.com/torvalds/linux/commit/2f2bd7cbd1d1548137b351040dc4e037d18... [3] https://github.com/torvalds/linux/commit/43527a0094c10dfbf0d5a2e7979395a38de...
The regression is that a middle click is performed when releasing middle button after wheel emulation.
The bug appears randomly, it can be after 5 minutes or 1 hour of keyboard usage, and can only be worked around by unplugging/re-plugging the keyboard. (I ended up resorting to simulate an unplug/replug, with a script which echoes 0 then 1 to /sys/bus/usb/devices/<id>/authorized, since I was afraid to damage the Micro-USB outlet by physically unplugging/re-plugging too much).
Those spurious clicks are very annoying, since they can open links in new tabs when scrolling in Firefox, or pasting text when scrolling in terminals, or other unwanted stuff.
I witnessed it with latest kernels (Debian unstable) as well as stable kernels (Debian 12 Bookworm, stable).
On Debian Stable, the last working kernel was 5.10.127, the regression appeared in 5.10.136 (i read all changelogs on kernel.org between those two releases but couldn't find anything about hid-lenovo, so I can't tell exactly in which release the regression appeared, Debian upgraded directly from .127 to .136).
I reported it in Debian [4], and apparently I'm not the only person suffering from it [5].
[4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#32 [5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#42
I would understand that such bugs would end up in a development kernel like the ones provided by Debian Unstable, but not with stable kernels like the ones provided by Debian Stable.
Regards,
[CCing the regression list, as it should be in the loop for regressions: https://docs.kernel.org/admin-guide/reporting-regressions.html]
Hi, Thorsten here, the Linux kernel's regression tracker.
On 16.02.24 12:51, Raphaël Halimi wrote:
(sorry for the long CC list, it looks quite long to me, but I tried to follow the issue reporting guide as closely as possible)
Since patches [1], [2] and [3] were applied to the kernel, there is a regression with Lenovo ThinkPad Compact USB Keyboard (old model, not II).
[1] https://github.com/torvalds/linux/commit/46a0a2c96f0f47628190f122c2e3d879e59... [2] https://github.com/torvalds/linux/commit/2f2bd7cbd1d1548137b351040dc4e037d18... [3] https://github.com/torvalds/linux/commit/43527a0094c10dfbf0d5a2e7979395a38de...
The regression is that a middle click is performed when releasing middle button after wheel emulation.
How did you identify these three commits? Or do you just suspect that it's one of them?
And did you try to check which of the three is the actual culprit? Either by reverting them on top of master or by checking the parent for each of the commits (git show '2f2bd7cbd1d^' shows the parent for 2f2bd7cbd1d).
The bug appears randomly, it can be after 5 minutes or 1 hour of keyboard usage, and can only be worked around by unplugging/re-plugging the keyboard. (I ended up resorting to simulate an unplug/replug, with a script which echoes 0 then 1 to /sys/bus/usb/devices/<id>/authorized, since I was afraid to damage the Micro-USB outlet by physically unplugging/re-plugging too much).
Those spurious clicks are very annoying, since they can open links in new tabs when scrolling in Firefox, or pasting text when scrolling in terminals, or other unwanted stuff.
I witnessed it with latest kernels (Debian unstable) as well as stable kernels (Debian 12 Bookworm, stable).
On Debian Stable, the last working kernel was 5.10.127, the regression appeared in 5.10.136 (i read all changelogs on kernel.org between those two releases but couldn't find anything about hid-lenovo, so I can't tell exactly in which release the regression appeared, Debian upgraded directly from .127 to .136).
Why not bisect between .127 and .136 then?
I reported it in Debian [4], and apparently I'm not the only person suffering from it [5].
[4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#32 [5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#42
I would understand that such bugs would end up in a development kernel like the ones provided by Debian Unstable, but not with stable kernels like the ones provided by Debian Stable.
A bug report like yours can do the trick sometimes, as it might be enough to ring a bell for one of the developers. But given that nobody replied yet it looks like that is not the case. Then you most likely will need to perform a bisection to identify the exact commit that broke things.
FWIW, I'm currently working on a new document describing the bisection, maybe it's of help for you: https://www.leemhuis.info/files/misc/How%20to%20bisect%20a%20Linux%20kernel%...
Ciao, Thorsten
P.S.: To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:
#regzbot ^introduced v5.10.127..v5.10.136 #regzbot title HID: lenovo: Lenovo ThinkPad Compact USB Keyboard sometimes sends middle-click #regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you.
Le 20/02/2024 à 12:35, Thorsten Leemhuis a écrit :
[CCing the regression list, as it should be in the loop for regressions: https://docs.kernel.org/admin-guide/reporting-regressions.html]
Hi, Thorsten here, the Linux kernel's regression tracker.
Hi, thanks for replying (even if I find your tone a bit harsh, but I don't blame you - and since English is not my native language, maybe I'm mistaking).
[1] https://github.com/torvalds/linux/commit/46a0a2c96f0f47628190f122c2e3d879e59... [2] https://github.com/torvalds/linux/commit/2f2bd7cbd1d1548137b351040dc4e037d18... [3] https://github.com/torvalds/linux/commit/43527a0094c10dfbf0d5a2e7979395a38de...
The regression is that a middle click is performed when releasing middle button after wheel emulation.
How did you identify these three commits? Or do you just suspect that it's one of them?
No, I didn't "just suspect" that it was one of them. I may not be a kernel developer but I'm an experienced sysadmin (25+ years). So please stop taking users for idiots.
First, I compared the three machines I used which have a keyboard with a TrackPoint: my desktop at home (external "Lenovo ThinkPad Compact Keyboard with TrackPoint" (not II, not Bluetooth), Debian unstable (I'm a DM), my desktop at work (same keyboard, Debian Stable) and my personal laptop (ThinkPad X270, internal keyboard, Debian Stable but with backports).
The machine at work had a 5.10 kernel at the time, and the other ones had a 6.6, but only the machines with an external keyboard exhibited the spurious middle-clicks. So I compared the loaded HID drivers, and noticed that both of them had hid_lenovo loaded, whereas the laptop did not.
Confident that I probably pinpointed the faulty driver, I simply looked at the file history on Github, and saw that those three commits were dated from after the time when the bug appeared ; moreover, the comments did mention stuff related to wheel emulation and spurious middle-clicks.
So, no, I didn't "just suspected" that they were responsible, but I hope you'll admit my method was sound, and that my conclusion is a pretty strong (to not say "almost certain") probability.
And did you try to check which of the three is the actual culprit? Either by reverting them on top of master or by checking the parent for each of the commits (git show '2f2bd7cbd1d^' shows the parent for 2f2bd7cbd1d).
I admit I didn't. I didn't compile my own kernels for ages. I used to do it in the past, but I came to trust Debian's kernels and rely on the maintainers' work. But read below.
On Debian Stable, the last working kernel was 5.10.127, the regression appeared in 5.10.136 (i read all changelogs on kernel.org between those two releases but couldn't find anything about hid-lenovo, so I can't tell exactly in which release the regression appeared, Debian upgraded directly from .127 to .136).
Why not bisect between .127 and .136 then?
I heard of that term before (and I understand the mathematical meaning of it), but I never did it with a Git tree. I read the guide you mentioned below, but it seems much too complicated and too long to me for just verifying if those three commits are indeed the cause of the regression (which I'm almost sure of, as stated above).
So in the meantime, I decided to follow my hunch and recompile only the hid_lenovo module (following the guide at [6], updating it slightly by manually removing kernel signing options in .config, since I obviously don't have Debian's signing keys, and replacing "make SUBDIRS=drivers/..." with "make M=..." as suggested by make), after un-applying those three patches in reverse order.
[6] https://askubuntu.com/a/338403/387067
The HID modules built successfully, and after copying my modified hid-lenovo.ko to /usr/lib/modules/6.6.15-amd64/updates/ and running 'depmod -a', the module loaded fine with Debian's kernel (I don't use Secure Boot on this machine).
I'll let a few days pass (remember, the bug doesn't happen immediately but only after a varying amount of time) and I'll report here if the spurious middle-clicks happened again or not.
Notes:
1/ Thank you for (indirectly) giving me this idea. Maybe this relatively simple procedure should be made available somewhere on Debian's wiki (instead of an outdated, but still useful, answer on AskUbuntu).
2/ Please note that I did it only for unstable kernel; unfortunately, I can't do the same for the stable kernel, since I don't have access to my machine at work anymore (my freelance contract ended one week ago) and I don't have any other machine at home exhibiting this bug. So I won't be able to test it on a stable kernel.
I reported it in Debian [4], and apparently I'm not the only person suffering from it [5].
[4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#32 [5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058758#42
I would understand that such bugs would end up in a development kernel like the ones provided by Debian Unstable, but not with stable kernels like the ones provided by Debian Stable.
A bug report like yours can do the trick sometimes, as it might be enough to ring a bell for one of the developers. But given that nobody replied yet it looks like that is not the case. Then you most likely will need to perform a bisection to identify the exact commit that broke things.
Nobody amongst the developers, yes, I'll give you that. But the comment I linked from the Debian BTS, plus another bug report I found in the Input mailing list [7], show that I'm not the only user complaining from the recent regressions.
[7] https://lore.kernel.org/linux-input/CACSVgagaEHO2zoYQ8zDBrMT9OvT8R5B_h3dxfZu...
Regards,
Le 20/02/2024 à 19:12, Raphaël Halimi a écrit :
Confident that I probably pinpointed the faulty driver, I simply looked at the file history on Github, and saw that those three commits were dated from after the time when the bug appeared ; moreover, the comments did mention stuff related to wheel emulation and spurious middle-clicks.
s/after/just before/
Regards,
Le 20/02/2024 à 19:12, Raphaël Halimi a écrit :
I'll let a few days pass (remember, the bug doesn't happen immediately but only after a varying amount of time) and I'll report here if the spurious middle-clicks happened again or not.
As promised, here's my report: using the recompiled hid-lenvo module without those three patches for more than three days, I didn't experience a single spurious middle-click, whereas the in-tree module triggered the bug several times a day, and I had to unplug/replug the keyboard (or simulate it with a software trick) to get back to a normal state.
So those three patches did introduce this regression after all (as I correctly guessed).
Regards,
On 24.02.24 11:52, Raphaël Halimi wrote:
Le 20/02/2024 à 19:12, Raphaël Halimi a écrit :
I'll let a few days pass (remember, the bug doesn't happen immediately but only after a varying amount of time) and I'll report here if the spurious middle-clicks happened again or not.
As promised, here's my report: using the recompiled hid-lenvo module without those three patches for more than three days, I didn't experience a single spurious middle-click, whereas the in-tree module triggered the bug several times a day, and I had to unplug/replug the keyboard (or simulate it with a software trick) to get back to a normal state.
So those three patches did introduce this regression after all (as I correctly guessed).
Mikhail, do you have any idea what might be wrong here? The three commits Raphaël mentioned that seem to cause the issue are all yours afaics.
Raphaël: would nevertheless still be good if you could identify which of the three causes the problem, as then the developers might consider simply reverting it.
Ciao, Thorsten
Le 24/02/2024 à 14:08, Thorsten Leemhuis a écrit :
Raphaël: would nevertheless still be good if you could identify which of the three causes the problem, as then the developers might consider simply reverting it.
Hi,
It can't be the third one (43527a0) since I clearly remember that I experienced the regression before it was applied to the Debian kernel.
So I'll try applying only the first one (46a0a2c), and report.
(in the meantime I crafted a quick and dirty Debian package to build the module with DKMS, so it will be easy)
Regards,
Le 24/02/2024 à 14:51, Raphaël Halimi a écrit :
It can't be the third one (43527a0) since I clearly remember that I experienced the regression before it was applied to the Debian kernel.
So I'll try applying only the first one (46a0a2c), and report.
I can confirm that the module compiled with 46a0a2c alone does produces spurious middle-clicks.
Maybe "ThinkPad Compact Keyboard with TrackPoint" should also be excluded, like "ThinkPad TrackPoint Keyboard II" was in commit 43527a0 ?
But then, would 46a0a2c still be relevant ?
Regards,
On 24.02.24 17:15, Raphaël Halimi wrote:
Le 24/02/2024 à 14:51, Raphaël Halimi a écrit :
It can't be the third one (43527a0) since I clearly remember that I experienced the regression before it was applied to the Debian kernel.
So I'll try applying only the first one (46a0a2c), and report.
I can confirm that the module compiled with 46a0a2c alone does produces spurious middle-clicks.
Maybe "ThinkPad Compact Keyboard with TrackPoint" should also be excluded, like "ThinkPad TrackPoint Keyboard II" was in commit 43527a0 ?
But then, would 46a0a2c still be relevant ?
Hmmm, another week without any developer looking at this. That's not how it should be. Guess I have to bring this to Linus attention sooner or later then. But before doing so, please confirm that 6.8-rc8 is still affected and reverting the culprit on top of it fixes the problem (the tricks you used are not bad as such, but they can have side effects -- which might also be the reason why no developer has looked into this).
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr If I did something stupid, please tell me, as explained on that page.
#regzbot poke
Hi,
Sorry for ignoring this thread. I've submitted the fix [1] quite a while ago but it is now in hid tree targeting 6.9. Maybe we can redirect it into 6.8? Jiri, what do you think?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/hid/hid.git/commit/?h=for-6....
On Mon, 4 Mar 2024 at 16:34, Linux regression tracking (Thorsten Leemhuis) regressions@leemhuis.info wrote:
On 24.02.24 17:15, Raphaël Halimi wrote:
Le 24/02/2024 à 14:51, Raphaël Halimi a écrit :
It can't be the third one (43527a0) since I clearly remember that I experienced the regression before it was applied to the Debian kernel.
So I'll try applying only the first one (46a0a2c), and report.
I can confirm that the module compiled with 46a0a2c alone does produces spurious middle-clicks.
Maybe "ThinkPad Compact Keyboard with TrackPoint" should also be excluded, like "ThinkPad TrackPoint Keyboard II" was in commit 43527a0 ?
But then, would 46a0a2c still be relevant ?
Hmmm, another week without any developer looking at this. That's not how it should be. Guess I have to bring this to Linus attention sooner or later then. But before doing so, please confirm that 6.8-rc8 is still affected and reverting the culprit on top of it fixes the problem (the tricks you used are not bad as such, but they can have side effects -- which might also be the reason why no developer has looked into this).
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr If I did something stupid, please tell me, as explained on that page.
#regzbot poke
Le 04/03/2024 à 15:52, Mikhail Khvoinitsky a écrit :
Hi,
Sorry for ignoring this thread. I've submitted the fix [1] quite a while ago but it is now in hid tree targeting 6.9. Maybe we can redirect it into 6.8? Jiri, what do you think?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/hid/hid.git/commit/?h=for-6....
I'd be glad to test the module with this patch applied.
What's the default setting ? Should I set any parameter in sysfs to get the desired result (apply workaround) ?
Regards,
I'd be glad to test the module with this patch applied.
Sure.
What's the default setting ? Should I set any parameter in sysfs to get the desired result (apply workaround) ?
Default is 1, so you don't have to change anything.
On Mon, 4 Mar 2024 at 17:07, Raphaël Halimi raphael.halimi@gmail.com wrote:
Le 04/03/2024 à 15:52, Mikhail Khvoinitsky a écrit :
Hi,
Sorry for ignoring this thread. I've submitted the fix [1] quite a while ago but it is now in hid tree targeting 6.9. Maybe we can redirect it into 6.8? Jiri, what do you think?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/hid/hid.git/commit/?h=for-6....
I'd be glad to test the module with this patch applied.
What's the default setting ? Should I set any parameter in sysfs to get the desired result (apply workaround) ?
Regards,
-- Raphaël Halimi
Le 04/03/2024 à 16:12, Mikhail Khvoinitsky a écrit :
I'd be glad to test the module with this patch applied.
Sure.
What's the default setting ? Should I set any parameter in sysfs to get the desired result (apply workaround) ?
Default is 1, so you don't have to change anything.
Thanks, it's done. I'll test and report.
Regards,
Le 04/03/2024 à 17:09, Raphaël Halimi a écrit :
Thanks, it's done. I'll test and report.
Nearly a week testing this patch (with kernels 6.6.15, 6.7.7 and 6.7.9, following Debian unstable updates) and it's working well so far.
Not a single spurious middle-click, which is not surprising since, if I understand correctly, this last patch just disables 46a0a2c and makes it optional, allowing to enable it on demand with a setting in sysfs.
And I have vertical and horizontal scrolling with the middle button working reliably (I'm not sure of what you mean by "hi-res scrolling", is it about 4K displays ?).
So as far as I'm concerned, this patch should be included ASAP in the next kernels releases (both latest and stable).
Regards,
Not a single spurious middle-click, which is not surprising since, if I understand correctly, this last patch just disables 46a0a2c and makes it optional, allowing to enable it on demand with a setting in sysfs.
That's correct.
And I have vertical and horizontal scrolling with the middle button working reliably
If you mean my statement in the initial commit that the original firmware doesn't support horizontal scrolling, I might be wrong, looks like I've mixed it up with something. But the main reason for the change was hi-res scrolling.
(I'm not sure of what you mean by "hi-res scrolling", is it about 4K displays ?).
No, it's about scrolling not by a fixed amount of lines but by individual pixels depending on how strongly you press the trackpoint. More like modern touchpads work.
So as far as I'm concerned, this patch should be included ASAP in the next kernels releases (both latest and stable).
Yes, as soon as it gets into master (given that 6.8 has just been released it will be soon), I'll make sure it will be included in stable (either automatically or manually).
On Tue, 12 Mar 2024 at 13:57, Raphaël Halimi raphael.halimi@gmail.com wrote:
Le 04/03/2024 à 17:09, Raphaël Halimi a écrit :
Thanks, it's done. I'll test and report.
Nearly a week testing this patch (with kernels 6.6.15, 6.7.7 and 6.7.9, following Debian unstable updates) and it's working well so far.
Not a single spurious middle-click, which is not surprising since, if I understand correctly, this last patch just disables 46a0a2c and makes it optional, allowing to enable it on demand with a setting in sysfs.
And I have vertical and horizontal scrolling with the middle button working reliably (I'm not sure of what you mean by "hi-res scrolling", is it about 4K displays ?).
So as far as I'm concerned, this patch should be included ASAP in the next kernels releases (both latest and stable).
Regards,
-- Raphaël Halimi
Le 12/03/2024 à 14:05, Mikhail Khvoinitsky a écrit :
(I'm not sure of what you mean by "hi-res scrolling", is it about 4K displays ?).
No, it's about scrolling not by a fixed amount of lines but by individual pixels depending on how strongly you press the trackpoint. More like modern touchpads work.
I didn't even know that the TrackPoint was pressure-sensitive :) I quickly tested this (not with scrolling, only cursor movement) and indeed, if I apply stronger pressure, the cursor moves faster. I never noticed that. We learn something everyday.
So as far as I'm concerned, this patch should be included ASAP in the next kernels releases (both latest and stable).
Yes, as soon as it gets into master (given that 6.8 has just been released it will be soon), I'll make sure it will be included in stable (either automatically or manually).
Perfect. Thank you for your work !
Regards,
linux-stable-mirror@lists.linaro.org
-
Linux regression tracking (Thorsten Leemhuis) -
Mikhail Khvoinitsky -
Mikhail Khvoinitsky -
Raphaël Halimi -
Thorsten Leemhuis