Dear Linux Stable Maintainers,
while maintaining downstream Linux releases, we noticed that we have to backport some patches manually, because they are not picked up by your automated backporting. Some of these backports can be done with improved cherry-pick tooling. We have implemented a script/tool "git- fuzzy-pick" which we would like to share. Besides picking more commits, the tool also supports executing a validation script right after picking, e.g. compiling the modified source file. Picking stats and details are presented below.
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
To test backporting performance, we tried to backport stable-candidate patches from 6.12 to 6.1. Specifically, on tag 6.1.125 we executed the command stable show-missing-stable v6.12.12..v6.12.17 to collect patches considered for backporting. This results in 431 backport candidates. When using git-fuzzy-pick, we can pick 9 patches more than with default cherry-picking. All modifications have been validated by attempting to build the object files of the modified C source files with make using the kernels “allyesconfig” configuration.
196 Cherry-picked with --strategy=recursive --Xpatience -x 1 Applied with patch -p1 ... --fuzz=1 8 Applied with patch -p1 ... --fuzz=2
Please let us know how to best share the tool with you! Long term, we would like to integrate it into your backporting workflow, so that more kernel commits can be applied automatically.
Best, Norbert
[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/sashal/stable-tools...
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
Dear Linux Stable Maintainers,
while maintaining downstream Linux releases, we noticed that we have to backport some patches manually, because they are not picked up by your automated backporting. Some of these backports can be done with improved cherry-pick tooling. We have implemented a script/tool "git- fuzzy-pick" which we would like to share. Besides picking more commits, the tool also supports executing a validation script right after picking, e.g. compiling the modified source file. Picking stats and details are presented below.
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
To test backporting performance, we tried to backport stable-candidate patches from 6.12 to 6.1. Specifically, on tag 6.1.125 we executed the command stable show-missing-stable v6.12.12..v6.12.17 to collect patches considered for backporting. This results in 431 backport candidates. When using git-fuzzy-pick, we can pick 9 patches more than with default cherry-picking. All modifications have been validated by attempting to build the object files of the modified C source files with make using the kernels “allyesconfig” configuration.
196 Cherry-picked with --strategy=recursive --Xpatience -x 1 Applied with patch -p1 ... --fuzz=1 8 Applied with patch -p1 ... --fuzz=2
Please let us know how to best share the tool with you! Long term, we would like to integrate it into your backporting workflow, so that more kernel commits can be applied automatically.
A long long time ago I used to apply patches with a whole lot of fuzz in a semi-automated way much like this, but things slipped through all the time. So now, if we have a failure, I throw it back to the developers/maintainers involved and let them and/or the community deal with it if they want to have the commit backported. That way they can test and manually verify that the backport is correct.
So no, I don't recommend auto-fuzz-forcing any commits WITHOUT manually looking and verifying and checking and taking responsibility for the backport being correct. Right now we are full of work just keeping up with 40 commits a day in the stable trees. If others wish to help out, we are more than willing to take the backports that have been submitted to us, and we now can handle them directly from the mailing lists even easier than before, with Sasha's bot reporting any potential problems, and my local scripts that properly apply them to the specific queues in a "very few" key-stroke command from my email client.
So in summary, I'm more than willing to take a patch that adds your tool to our repo, for others to use, but I don't want to be the one responsible for running it. I want others to take that responsibility if they care about applying those patches to older kernels properly, as it's the companies that care about those kernels that should be the ones helping us out the most, instead of asking others to do this work for them, don't you think? :)
thanks,
greg k-h
On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
Dear Linux Stable Maintainers,
while maintaining downstream Linux releases, we noticed that we have to backport some patches manually, because they are not picked up by your automated backporting. Some of these backports can be done with improved cherry-pick tooling. We have implemented a script/tool "git- fuzzy-pick" which we would like to share. Besides picking more commits, the tool also supports executing a validation script right after picking, e.g. compiling the modified source file. Picking stats and details are presented below.
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
Wait, you should be looking at the scripts in the stable-queue.git tree in the scripts/ directory. You pointed at a private repo of some things that Sasha uses for his work, which is specific to his workflow.
thanks,
greg k-h
On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
Dear Linux Stable Maintainers,
while maintaining downstream Linux releases, we noticed that we have to backport some patches manually, because they are not picked up by your automated backporting. Some of these backports can be done with improved cherry-pick tooling. We have implemented a script/tool "git- fuzzy-pick" which we would like to share. Besides picking more commits, the tool also supports executing a validation script right after picking, e.g. compiling the modified source file. Picking stats and details are presented below.
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
Wait, you should be looking at the scripts in the stable-queue.git tree in the scripts/ directory. You pointed at a private repo of some things that Sasha uses for his work, which is specific to his workflow.
Also, one final things. Doing backports to older kernels is a harder task than doing it for newer kernels. This means you need to do more work, and have a more experienced developer do that work, as the nuances are tricky and slight and they must understand the code base really well.
Attempting to automate this, and make it a "junior developer" task assignment is ripe for errors and problems and tears (on my side and yours.) We have loads of examples of this in the past, please don't duplicate the errors of others and think that "somehow, this time it will be different!", but rather "learn from our past mistakes and only make new ones."
Good luck with backporting, as I know just how hard of a task this really is. And obviously, you are learning that too :)
greg k-h
On Thu, 2025-04-03 at 15:51 +0100, gregkh@linuxfoundation.org wrote: ...snip ...
On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
...snip...
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
Thanks!
Wait, you should be looking at the scripts in the stable-queue.git tree in the scripts/ directory. You pointed at a private repo of some things that Sasha uses for his work, which is specific to his workflow.
I had a look at those scripts too. Looks like you use git am, and abort in case this operation fails.
Also, one final things. Doing backports to older kernels is a harder task than doing it for newer kernels. This means you need to do more work, and have a more experienced developer do that work, as the nuances are tricky and slight and they must understand the code base really well.
Attempting to automate this, and make it a "junior developer" task assignment is ripe for errors and problems and tears (on my side and yours.) We have loads of examples of this in the past, please don't duplicate the errors of others and think that "somehow, this time it will be different!", but rather "learn from our past mistakes and only make new ones."
We understand. We might make the tool available to help simplify the human effort of backporting. To make this more successful, is there a way to identify the errors and learnings you mention from the past? Avoiding them automatically early in the process helps keeping the errors away.
Good luck with backporting, as I know just how hard of a task this really is. And obviously, you are learning that too :)
Thanks.
Norbert
On Mon, Apr 07, 2025 at 07:26:59AM +0000, Manthey, Norbert wrote:
On Thu, 2025-04-03 at 15:51 +0100, gregkh@linuxfoundation.org wrote: ...snip ...
On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
...snip...
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
Thanks!
Wait, you should be looking at the scripts in the stable-queue.git tree in the scripts/ directory. You pointed at a private repo of some things that Sasha uses for his work, which is specific to his workflow.
I had a look at those scripts too. Looks like you use git am, and abort in case this operation fails.
We only use 'git am' to apply the quilt series to the tree when doing a release (the final one or the -rc ones). We use quilt to manage everything in the stable-queue.git tree as that provides us with the needed flexibility.
Also, one final things. Doing backports to older kernels is a harder task than doing it for newer kernels. This means you need to do more work, and have a more experienced developer do that work, as the nuances are tricky and slight and they must understand the code base really well.
Attempting to automate this, and make it a "junior developer" task assignment is ripe for errors and problems and tears (on my side and yours.) We have loads of examples of this in the past, please don't duplicate the errors of others and think that "somehow, this time it will be different!", but rather "learn from our past mistakes and only make new ones."
We understand. We might make the tool available to help simplify the human effort of backporting. To make this more successful, is there a way to identify the errors and learnings you mention from the past? Avoiding them automatically early in the process helps keeping the errors away.
Don't ignore fuzz, manually check, and verify, everything.
good luck!
greg k-h
On Mon, Apr 07, 2025 at 09:31:08AM +0200, gregkh@linuxfoundation.org wrote:
We understand. We might make the tool available to help simplify the human effort of backporting. To make this more successful, is there a way to identify the errors and learnings you mention from the past? Avoiding them automatically early in the process helps keeping the errors away.
Don't ignore fuzz, manually check, and verify, everything.
Also, diffing the file between the latest kernel and the one you're backporting to helps discover changed assumptions. For example, a group of functions might be called with similar assumptions in the old kernel, with some tests replicated everywhere (say pointer foo must not be NULL). In newer kernels this test is moved up in the caller chain and is no longer performed in the lower functions. When you want to backport a fix from this kernel to the old one, you may need to reimplement yourself the nullity check that the old kernel requires. And that's valid for locking and many other things in general. There's no way to automatically discover these ones, aside comparing the older and newer states of the file to see how it evolved over time, and developing your own habits of remembering that certain areas are different in your kernel, by doing lots of backports there, as well as following LKML to try to spot some changes that may affect your areas of interest. After a few years on old kernel you can start to develop some reflexes but that's not rocket science.
good luck!
Seconded!
greg k-h
Willy
On Thu, Apr 03, 2025 at 03:51:25PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
Dear Linux Stable Maintainers,
while maintaining downstream Linux releases, we noticed that we have to backport some patches manually, because they are not picked up by your automated backporting. Some of these backports can be done with improved cherry-pick tooling. We have implemented a script/tool "git- fuzzy-pick" which we would like to share. Besides picking more commits, the tool also supports executing a validation script right after picking, e.g. compiling the modified source file. Picking stats and details are presented below.
We would like to discuss whether you can integrate this improved tool into into your daily workflows. We already found the stable-tools repository [1] with some scripts that help automate backporting. To contribute git-fuzzy-pick there, we would need you to declare a license for the current state of this repository.
There's no need for us to declare the license for the whole repo, you just need to pick a license for your script to be under. Anything that's under a valid open source license is fine with me.
That being said, I did just go and add SPDX license lines to all of the scripts that I wrote, or that was already defined in the comments of the files, to make it more obvious what they are under.
Wait, you should be looking at the scripts in the stable-queue.git tree in the scripts/ directory. You pointed at a private repo of some things that Sasha uses for his work, which is specific to his workflow.
Also, one final things. Doing backports to older kernels is a harder task than doing it for newer kernels. This means you need to do more work, and have a more experienced developer do that work, as the nuances are tricky and slight and they must understand the code base really well.
Attempting to automate this, and make it a "junior developer" task assignment is ripe for errors and problems and tears (on my side and yours.) We have loads of examples of this in the past, please don't duplicate the errors of others and think that "somehow, this time it will be different!", but rather "learn from our past mistakes and only make new ones."
Good luck with backporting, as I know just how hard of a task this really is. And obviously, you are learning that too :)
I've played with wiggle[1] in the past, which does all what you've described and more.
It introduced too many issues, where it wasn't worth doing it. I really think that the better solution here is to figure out dependencies and bring them in.
I'd advise against using such tool at the scale of -stable :)
[1] https://github.com/neilbrown/wiggle
linux-stable-mirror@lists.linaro.org
-
gregkh＠linuxfoundation.org -
Manthey, Norbert -
Sasha Levin -
Willy Tarreau