On 14/10/2023 11:43, Willy Tarreau wrote:
Hi Vegard,
On Fri, Oct 13, 2023 at 05:24:31PM +0200, Vegard Nossum wrote:
I've now added Steven Rostedt and Willy Tarreau as well on the off-chance that they have something to say about it (Steven presented his conflict resolution method at Kernel Recipes and I think Willy is experienced with backporting), but this is in no way meant as pressure to review this patch. Here's a link to the top of the thread:
https://lore.kernel.org/all/20230824092325.1464227-1-vegard.nossum@oracle.co...
(Adding Ben Hutchings to Cc as well for the same reasons.)
That's a very nice description, I'm sure it can help (and I learned a few points there already). There are a few points I'm not seeing there, though, base on my habits:
Thanks for the quick and comprehensive response!
- in my experience, there's a huge difference between backporting code you don't know and code you know. I'm dealing with haproxy backports several times a week and I tend to know this code, so I use my intuition a lot and have no problem adjusting stuff around the conflicting parts. However when I was dealing with extended kernels, that was not the case at all, because I didn't know that code, and worse, I wasn't skilled at all on many of the parts I had to deal with. While it's OK to take the blame for a failed backport, it's generally not OK to expose users to risks caused by your lack of knowledge. In this case it means you need to be extra cautious, and very often to actually *ask* authors or maintainers for help. If maintainers do not want to help backporting some patches to an older version of their code, usually it should be perceived as a hint that they'll find it complicated to do it right; then take that as a hint that there's little chances you'll get it right by yourself while ignoring that code. This may imply dropping the fix, documenting the area as broken, or asking for help on various lists until someone more knowledgeable can help.
I agree -- backports ARE very easy to get wrong, EVEN when you know the code well. This is stressed several times in the document, especially in the last two sections about build and runtime testing, but also in the section on error handling.
However, I'm wary of being too stern as well. There are already a million ways to introduce subtle bugs and put users at risk, but we rarely try to put people off contributing regular patches (at least in this specific way :-P).
Did you see this meme? https://i.imgur.com/yk5rf13.png
I think conflicts have a bit of a bad reputation exactly because you're presented with something that's hard to make sense of at first sight. I'd like to dispel this myth that you need to be an expert to make sense of conflict markers. I think with the right attitude, the right tools, and the right approach you can go a LONG way. Also, nobody is born an expert and we should encourage people to gain experience in this area IMHO.
With that said, how about if we add a short section near the end about submitting stable backports where we encourage submitters to 1) approach the backporting process in a humble way, 2) describe the reason for the conflict in the changelog and their resolution, 3) be honest about their confidence in their resolution, 4) ask relevant maintainers for an explicit ack?
I'm open to other ideas, I just want to make sure we strike the right balance of encouragement vs. discouragement.
the tool that helped me the most in resolving rename conflicts is "patch". As you explained, "git am" is a bit stubborn. But patch is way more lenient (and will often do mistakes). In the very old 2.6.32 I used to rely on "git show XX | patch -p1" way more often than "git am". For a rename, you do "git show XX -- file |patch otherfile". It works the same with file-based patches or mbox: "patch -p1 < mbox". Patch however will not place the conflict markers and will leave .rej files. I then opened them in an editor next to the file to edit, to locate the area and copy the relevant part to its location. Emacs' ediff is also extremely convenient to pick select parts of each file.
control the patches: after I'm pretty sure I have resolved a patch, I compare it side by side with the original one. Normally, backported patches should have the same structure as the original. Using whatever editor supporting a vertical split helps a lot. Otherwise I also use "diff -y --width 200" between them to focus on differences (typically line numbers). It's not rare that a part is missing, either because I messed up, or because I forgot to process a .rej that I mistakenly removed, or because a file was added, I reset the tree and it's left untracked. So any difference between the patches should have its own explanation (line numbers, function names, file names, occurrences). By the way, it can very easily happen that applying a patch will work fine but it will apply to the wrong function because some code patterns especially in error unrolling paths are often the same between several functions. It happened to me quite a few times in the past, and even a few of these persisted till the public patch review. That's really a risk that must not be minimized!
There is a section on this: "Verifying the result", and also describes doing a side-by-side diff of the old and new patches.
The bit about applying the patch to the wrong function -- I doubt this happens that much when using cherry-pick, since it knows both sides of the history and can tell when code moves around. You're probably right that it can easily happen with plain git am/patch though. In my mind, this is an argument in favour of using cherry-pick whenever possible.
- something quite common is that as code evolves, it gets refactored so that what used to appear at 3 locations in the past now moved to a single function. But when you're backporting, you're doing the reverse work, you're taking a patch for a single location that may apply to multiple ones. Often the hint is that the function name changed. But sometimes it's not even the case. If what you've found looks like a nasty bug pattern that looks like it could easily have been copy-pasted at other places, it's worth looking for it elsewhere using git grep. If you find the same pattern, then you search for it into the tree the patch comes from. If you don't find it, it's likely that you'll need to adjust it, and git log is your friend to figure what happened to these areas. Note that git blame doesn't work for removed code. If you find the same pattern elsewhere in mainline, it's worth asking the patch author if that one is also an occurrence of the same bug or just normal. It's not uncommon to find new bugs during a backport.
Very good point.
I think this fits very well alongside the sections on function arguments, error handling, etc. since it details a specific thing that can go easily wrong.
Can I take what you wrote above, or do you want to submit your own incremental patch? I think we could insert it almost verbatim.
color diff: usually I just rely on:
[color] ui = true
But I also recently got used to using diff-highlight that will highlight different characters between lines. It's nice for complex "if" conditions where you don't see the difference, or for spaces vs tabs:
[pager] log = /usr/doc/git-2.35.3/contrib/diff-highlight/diff-highlight | less show = /usr/doc/git-2.35.3/contrib/diff-highlight/diff-highlight | less diff = /usr/doc/git-2.35.3/contrib/diff-highlight/diff-highlight | less
Right, git log/show/diff --word-diff=color can also do this to some degree.
There is also core.whitespace/color.diff.whitespace that will highlight some common whitespace errors.
I haven't used diff-highlight myself -- I'll give it a try.
In this case, I was using colordiff specifically to do the side-by-side diff of the two patches since it handles the <() shell syntax _and_ does the highlighting of differences between the patches.
- git add, git add and git add: when fixing patches by hand, it's very common to leave some parts not added (especially with | patch -p1). It's useful to work on a clean tree to more easily spot untracked files with "git status".
Yet another reason to use git cherry-pick instead of manual patch commands: it keeps track of unmerged files for you. ;-)
So I'm a bit torn on this. I think in this particular document I'd like to encourage the use of git and doing things "the git way" as much as possible.
I feel like in the worst case, somebody sees the document down the line and vehemently disagrees with something and we either fix it or take it out completely.
No I don't disagree and even find it useful. If at least it could help people figure the pain it is to backport any single patch, and encourage them to help stable maintainers, that would already be awesome!
I'd like to add that my impression is that a LOT of people *fear* backporting and conflict resolution -- and it doesn't have to be that way. We should be talking about merge conflicts and what good workflows look like (one of the reasons why I was very happy to see Steven's presentation at KR), instead of leaving everybody to figure it out on their own. This document is my contribution towards that.
I'm not completely sold to this. Yes we should teach more people to perform that task themselves. But there's a big difference between backporting a few patches and feeling like you could maintain your own kernel because now you know how to resolve conflicts. What I mentioned above about dealing with patches you don't understand must not be underestimated, that's the biggest challenge I faced when working on stable kernels. There's probably a feeling of shame of not understanding something, but I can say that many times I asked for help and was helped even by top-ranked developers, and nobody ever laughed at me for not understanding a certain area. But doing that in your garage for your own kernel or for your company's products is a huge problem because it's unlikely that you'll get help from the maintainers this time, so you're left on your own with your own understanding of certain patches.
Thus, yes to backports, no to kernel forks being a collection of backports.
Right; almost every time I talk about backporting it's really in the context of contributing these backports to stable. I'm not in favour of forks either and I'm not trying to encourage it.
Let me try to come up with a specific addition related to the changes you requested above and see if you agree with the wording.
Thanks,
Vegard