Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed """
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
- Luiz
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from? What about the other stable releases?
thanks,
greg k-h
On 2023-06-01 02:06, Greg KH wrote:
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from?
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
What about the other stable releases?
I tested against: 6.1.31, 5.15.114, 5.10.181 they all reproduce it. I'd guess any kernel having a backport of 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 will have this.
One detail though, I tested once with "make binrpm-pkg" and didn't reproduce it. I think this could mean that it depends on the make targets you use to build the kernel. But "make bzImage", "make modules", "make modules_install" is pretty standard (or used to be).
- Luiz
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
On 2023-06-01 02:06, Greg KH wrote:
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from?
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
When this gets fixed in Linus's tree, I'll be glad to backport the changes to the other kernels. Please work with the developers to get that fixed there.
thanks,
greg k-h
On 2023-06-01 09:20, Greg KH wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
On 2023-06-01 02:06, Greg KH wrote:
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from?
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
When this gets fixed in Linus's tree, I'll be glad to backport the changes to the other kernels. Please work with the developers to get that fixed there.
OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?
- Luiz
On Thu, Jun 01, 2023 at 09:26:30AM -0400, Luiz Capitulino wrote:
On 2023-06-01 09:20, Greg KH wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
On 2023-06-01 02:06, Greg KH wrote:
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from?
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
When this gets fixed in Linus's tree, I'll be glad to backport the changes to the other kernels. Please work with the developers to get that fixed there.
OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?
We should avoid regressing Linus's kernel tree just as much. It's always been this way, I don't want to revert patches unless they are really bad if the fix is not in Linus's tree already. Gives people "encouragement" to resolve the issue quicker.
thanks,
greg k-h
On 2023-06-01 10:13, Greg KH wrote:
On Thu, Jun 01, 2023 at 09:26:30AM -0400, Luiz Capitulino wrote:
On 2023-06-01 09:20, Greg KH wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
On 2023-06-01 02:06, Greg KH wrote:
On Wed, May 31, 2023 at 10:12:40PM -0400, Luiz Capitulino wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
Then compare the version strings in the bzImage and vmlinux images, we can use "strings" for this. For example, in the 5.10.181 kernel I get:
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #2 SMP Thu Jun 1 01:26:38 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181 (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:23:59 UTC 2023
The version string in the bzImage doesn't have the "Linux version" part, but I think this is added by the kernel when printing. If you compare the strings, you'll see that they have a different build date and the "#1" and "#2" are different.
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
$ strings vmlinux | egrep '^Linux version' Linux version 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
$ strings ./arch/x86_64/boot/bzImage | egrep 'ld version' 5.10.181+ (ec2-user@ip-172-31-79-134.ec2.internal) (gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15), GNU ld version 2.29.1-31.amzn2) #1 SMP Thu Jun 1 01:29:11 UTC 2023
Maybe the grouped target usage in 4ce1f694eb5 with older "make" is causing a rebuild of the vmlinux image in "make modules"? If yes, is this expected?
I'm afraid this issue could be high impact for distros with older user-space.
Is this issue also in 6.4-rc1 where this change came from?
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
When this gets fixed in Linus's tree, I'll be glad to backport the changes to the other kernels. Please work with the developers to get that fixed there.
OK, I'll report it there, but shouldn't we avoid regressing -stable kernels?
We should avoid regressing Linus's kernel tree just as much. It's always been this way, I don't want to revert patches unless they are really bad if the fix is not in Linus's tree already. Gives people "encouragement" to resolve the issue quicker.
Fair enough. Paul is in this thread, let's see if he picks it up from here. If not, I'll report it on lkml.
- Luiz
On Thu, Jun 1, 2023 at 9:20 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
...
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
While I really don't want to go back into the old arguments about what does, and does not, get backported to -stable, I do want to ask if there is some way to signal to the -stable maintainers that a patch should not be backported? Anything coming from the LSM, SELinux, or audit trees that I believe should be backported is explicitly marked with a stable@vger CC, as documented in stable-kernel-rules.rst, however it is generally my experience that patches with a 'Fixes:' tag are generally pulled into the -stable releases as well.
I could start dropping the 'Fixes:' tag from non-stable tagged commits, but that's a step backwards in my opinion.
I could start replying to every -stable backport email notice, but that seems like a lot of unnecessary work for something that was never marked for -stable in the first place. I'm guessing it would also add some additional management/testing burden to the -stable folks as well.
-- paul-moore.com
On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
On Thu, Jun 1, 2023 at 9:20 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
...
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
While I really don't want to go back into the old arguments about what does, and does not, get backported to -stable, I do want to ask if there is some way to signal to the -stable maintainers that a patch should not be backported? Anything coming from the LSM, SELinux, or audit trees that I believe should be backported is explicitly marked with a stable@vger CC, as documented in stable-kernel-rules.rst, however it is generally my experience that patches with a 'Fixes:' tag are generally pulled into the -stable releases as well.
Really? Right now we HAVE to pick up the Fixes: tagged commits in those subsystems as you are missing lots of real fixes. I just quick looked and noticed 8cf0a1bc1287 ("capabilities: fix potential memleak on error path from vfs_getxattr_alloc()") which you should have tagged, right?
In fact, I've considered most of the LSM code as a "we never tag anything for stable so we rely on the Fixes: pickup to clean up after us" subsystem. We have many of those in the kernel, so you are in good company :)
I could start dropping the 'Fixes:' tag from non-stable tagged commits, but that's a step backwards in my opinion.
If a commit has fixes: why wouldn't it be ok for stable trees? That feels very odd to me.
Anyway, if you really want, yes, we can add you to the "list of subsystems we do not pick anything for except by explicit cc: stable marking" that we have, but note, that feels wrong to me based on the very low number of patches being tagged for these directories over time.
I could start replying to every -stable backport email notice, but that seems like a lot of unnecessary work for something that was never marked for -stable in the first place. I'm guessing it would also add some additional management/testing burden to the -stable folks as well.
We have a list, so if you really want it, we can add you to it. But can you point out any Fixes: commits that we backported that we shouldn't have? If so, why was a Fixes: tag on it?
thanks,
greg k-h
On Thu, Jun 1, 2023 at 11:51 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
On Thu, Jun 1, 2023 at 9:20 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
...
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
While I really don't want to go back into the old arguments about what does, and does not, get backported to -stable, I do want to ask if there is some way to signal to the -stable maintainers that a patch should not be backported? Anything coming from the LSM, SELinux, or audit trees that I believe should be backported is explicitly marked with a stable@vger CC, as documented in stable-kernel-rules.rst, however it is generally my experience that patches with a 'Fixes:' tag are generally pulled into the -stable releases as well.
Really?
Yes, really.
Right now we HAVE to pick up the Fixes: tagged commits in those subsystems as you are missing lots of real fixes.
This starts to bring us back to the old argument about what is appropriate for -stable, but I've been sticking as close as possible to what is documented in stable-kernel-rules.rst which (ignoring things like HW enablement) advises that only patches which fix build issues or "serious issues" should be considered for -stable. I consider every bug fix that goes into the LSM, SELinux, and audit trees to see if it meets those criteria, if it does I mark it with a -stable tag, if not I leave the -stable tag and ensure it carries a 'Fixes:' tag if it makes sense and an appropriate root-cause commit is identified.
We definitely have different opinions on where the -stable bug fix threshold lies. I am of the opinion that every -stable backport carries risk, and I consider that when deciding if a commit should be marked for -stable. I do not believe that every bug fix, or every commit with a 'Fixes:' tag, should be backported to -stable.
I just quick looked and noticed 8cf0a1bc1287 ("capabilities: fix potential memleak on error path from vfs_getxattr_alloc()") which you should have tagged, right?
Nope. That's a memleak for a weird corner case that isn't really going to be triggered in practice; it was found only by code inspection and requires code modification to exercise. See the discussion around the original revision for more information:
https://lore.kernel.org/linux-security-module/20221025104544.2298829-1-cuiga...
Can you explain why you HAD to pick up that commit?
In fact, I've considered most of the LSM code as a "we never tag anything for stable so we rely on the Fixes: pickup to clean up after us" subsystem. We have many of those in the kernel, so you are in good company :)
That would be amusing if it were correct, but your use of absolutes betrays you. Looking at code under security/ from v5.10 to current on I see 71 -stable tags as of today. It looks like the last one I marked was from October of 2022. We could deep dive more on this, but I think that's a waste of everyone's time.
I could start dropping the 'Fixes:' tag from non-stable tagged commits, but that's a step backwards in my opinion.
If a commit has fixes: why wouldn't it be ok for stable trees? That feels very odd to me.
Backports have risk, and in my opinion not every bug fix rises to the level of severity to sufficiently counter that risk. The -stable kernel documentation seems to support that; if you believe every bug fix should be backported to the -stable kernels I would suggest you update the documentation.
Anyway, if you really want, yes, we can add you to the "list of subsystems we do not pick anything for except by explicit cc: stable marking" that we have, but note, that feels wrong to me based on the very low number of patches being tagged for these directories over time.
That's great, I didn't know you maintained such a list. Please add the LSM, SELinux, and audit trees to the subsystems which are only backported when there is an explicit -stable tag. It is worth noting that this should not affect the -stable backport policy for other LSMs, e.g. AppArmor, although I can mention this list to the other LSM maintainers as they may want to be added.
On Thu, Jun 01, 2023 at 02:39:00PM -0400, Paul Moore wrote:
On Thu, Jun 1, 2023 at 11:51 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 10:56:24AM -0400, Paul Moore wrote:
On Thu, Jun 1, 2023 at 9:20 AM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 09:13:21AM -0400, Luiz Capitulino wrote:
...
Yes. I'm reporting this here because I'm more concerned with -stable kernels since they're more likely to be running on older user-space.
Yeah, we are bug-compatible! :)
While I really don't want to go back into the old arguments about what does, and does not, get backported to -stable, I do want to ask if there is some way to signal to the -stable maintainers that a patch should not be backported? Anything coming from the LSM, SELinux, or audit trees that I believe should be backported is explicitly marked with a stable@vger CC, as documented in stable-kernel-rules.rst, however it is generally my experience that patches with a 'Fixes:' tag are generally pulled into the -stable releases as well.
Really?
Yes, really.
Right now we HAVE to pick up the Fixes: tagged commits in those subsystems as you are missing lots of real fixes.
This starts to bring us back to the old argument about what is appropriate for -stable, but I've been sticking as close as possible to what is documented in stable-kernel-rules.rst which (ignoring things like HW enablement) advises that only patches which fix build issues or "serious issues" should be considered for -stable. I consider every bug fix that goes into the LSM, SELinux, and audit trees to see if it meets those criteria, if it does I mark it with a -stable tag, if not I leave the -stable tag and ensure it carries a 'Fixes:' tag if it makes sense and an appropriate root-cause commit is identified.
We definitely have different opinions on where the -stable bug fix threshold lies. I am of the opinion that every -stable backport carries risk, and I consider that when deciding if a commit should be marked for -stable. I do not believe that every bug fix, or every commit with a 'Fixes:' tag, should be backported to -stable.
Ok, I'll not argue here, but it feels like there is a lack of changes for some of these portions of the kernel that end up in stable kernels. I'll trust you on this.
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
thanks,
greg k-h
On Wed, Jun 28, 2023 at 2:33 PM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 02:39:00PM -0400, Paul Moore wrote:
...
We definitely have different opinions on where the -stable bug fix threshold lies. I am of the opinion that every -stable backport carries risk, and I consider that when deciding if a commit should be marked for -stable. I do not believe that every bug fix, or every commit with a 'Fixes:' tag, should be backported to -stable.
Ok, I'll not argue here, but it feels like there is a lack of changes for some of these portions of the kernel that end up in stable kernels. I'll trust you on this.
I don't know what to say here ... aside from the previously discussed difference of opinion regarding stable tags, we just haven't had many bad bugs in the LSM layer, SELinux, or audit the past few years. I'd like to claim better code, better review, yadda yadda but in reality I'm sure it's just dumb luck.
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
I've been trying to ensure that the files/directories entries in MAINTAINERS are current, so that is probably as good a place as any to pull that info. Do the stable tools use that info already? In other words, if we update the entries in MAINTAINERS should we also notify you guys, or will you get it automatically?
Regardless, here is a list:
* Audit include/asm-generic/audit_*.h include/linux/audit.h include/linux/audit_arch.h include/uapi/linux/audit.h kernel/audit* lib/*audit.c
* LSM layer security/ (NOTE: the individual sub-dirs under security/ belong to the individual LSMs, not the LSM layer)
* SELinux include/trace/events/avc.h include/uapi/linux/selinux_netlink.h scripts/selinux/ security/selinux/
On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
I've been trying to ensure that the files/directories entries in MAINTAINERS are current, so that is probably as good a place as any to pull that info. Do the stable tools use that info already? In other words, if we update the entries in MAINTAINERS should we also notify you guys, or will you get it automatically?
We do not use (or at least I don't, I can't speak for Sasha here, but odds are we should unify this now), the MAINTAINERS file for this, but rather a list like you provided below, thanks.
Regardless, here is a list:
- Audit
include/asm-generic/audit_*.h include/linux/audit.h include/linux/audit_arch.h include/uapi/linux/audit.h kernel/audit* lib/*audit.c
- LSM layer
security/ (NOTE: the individual sub-dirs under security/ belong to the individual LSMs, not the LSM layer)
So security/*.c would cover this, not below that, right?
- SELinux
include/trace/events/avc.h include/uapi/linux/selinux_netlink.h scripts/selinux/ security/selinux/
Looks good, thanks for this.
greg k-h
On Thu, Jun 29, 2023 at 4:43 AM Greg KH gregkh@linuxfoundation.org wrote:
On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
I've been trying to ensure that the files/directories entries in MAINTAINERS are current, so that is probably as good a place as any to pull that info. Do the stable tools use that info already? In other words, if we update the entries in MAINTAINERS should we also notify you guys, or will you get it automatically?
We do not use (or at least I don't, I can't speak for Sasha here, but odds are we should unify this now), the MAINTAINERS file for this, but rather a list like you provided below, thanks.
Fair enough, if we ever have any significant restructuring I'll try to remember to update the stable folks. Although I'm guessing such a change would likely end up being self-reporting anyway.
Regardless, here is a list:
- Audit
include/asm-generic/audit_*.h include/linux/audit.h include/linux/audit_arch.h include/uapi/linux/audit.h kernel/audit* lib/*audit.c
- LSM layer
security/ (NOTE: the individual sub-dirs under security/ belong to the individual LSMs, not the LSM layer)
So security/*.c would cover this, not below that, right?
Yes, that should work.
- SELinux
include/trace/events/avc.h include/uapi/linux/selinux_netlink.h scripts/selinux/ security/selinux/
Looks good, thanks for this.
Thanks for maintaining the exception list.
On Thu, Jun 29, 2023 at 11:55:12AM -0400, Paul Moore wrote:
On Thu, Jun 29, 2023 at 4:43 AM Greg KH gregkh@linuxfoundation.org wrote:
On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
I've been trying to ensure that the files/directories entries in MAINTAINERS are current, so that is probably as good a place as any to pull that info. Do the stable tools use that info already? In other words, if we update the entries in MAINTAINERS should we also notify you guys, or will you get it automatically?
We do not use (or at least I don't, I can't speak for Sasha here, but odds are we should unify this now), the MAINTAINERS file for this, but rather a list like you provided below, thanks.
Fair enough, if we ever have any significant restructuring I'll try to remember to update the stable folks. Although I'm guessing such a change would likely end up being self-reporting anyway.
Regardless, here is a list:
- Audit
include/asm-generic/audit_*.h include/linux/audit.h include/linux/audit_arch.h include/uapi/linux/audit.h kernel/audit* lib/*audit.c
- LSM layer
security/ (NOTE: the individual sub-dirs under security/ belong to the individual LSMs, not the LSM layer)
So security/*.c would cover this, not below that, right?
Yes, that should work.
- SELinux
include/trace/events/avc.h include/uapi/linux/selinux_netlink.h scripts/selinux/ security/selinux/
Looks good, thanks for this.
Thanks for maintaining the exception list.
Cool, it's maintained here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree...
if it's ever needed to be updated in the future.
thanks,
greg k-h
On Thu, Jun 29, 2023 at 12:07 PM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 29, 2023 at 11:55:12AM -0400, Paul Moore wrote:
On Thu, Jun 29, 2023 at 4:43 AM Greg KH gregkh@linuxfoundation.org wrote:
On Wed, Jun 28, 2023 at 07:33:27PM -0400, Paul Moore wrote:
So, can I get a directory list or file list of what we should be ignoring for the AUTOSEL and "Fixes: only" tools to be ignoring?
I've been trying to ensure that the files/directories entries in MAINTAINERS are current, so that is probably as good a place as any to pull that info. Do the stable tools use that info already? In other words, if we update the entries in MAINTAINERS should we also notify you guys, or will you get it automatically?
We do not use (or at least I don't, I can't speak for Sasha here, but odds are we should unify this now), the MAINTAINERS file for this, but rather a list like you provided below, thanks.
Fair enough, if we ever have any significant restructuring I'll try to remember to update the stable folks. Although I'm guessing such a change would likely end up being self-reporting anyway.
Regardless, here is a list:
- Audit
include/asm-generic/audit_*.h include/linux/audit.h include/linux/audit_arch.h include/uapi/linux/audit.h kernel/audit* lib/*audit.c
- LSM layer
security/ (NOTE: the individual sub-dirs under security/ belong to the individual LSMs, not the LSM layer)
So security/*.c would cover this, not below that, right?
Yes, that should work.
- SELinux
include/trace/events/avc.h include/uapi/linux/selinux_netlink.h scripts/selinux/ security/selinux/
Looks good, thanks for this.
Thanks for maintaining the exception list.
Cool, it's maintained here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree...
if it's ever needed to be updated in the future.
Noted, thanks.
On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino luizcap@amazon.com wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
...
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
Thanks Luiz, this is a fun one :/
Based on a quick search, it looks like the grouped target may be the cause, especially for older (pre-4.3) versions of make. Looking through the rest of the kernel I don't see any other grouped targets, and in fact the top level Makefile even mentions holding off on using grouped targets until make v4.3 is common/required.
I don't have an older userspace immediately available, would you mind trying the fix/patch below to see if it resolves the problem on your system? It's a cut-n-paste so the patch may not apply directly, but it basically just removes the '&' from the make rule, turning it into an old-fashioned non-grouped target.
diff --git a/security/selinux/Makefile b/security/selinux/Makefile index 0aecf9334ec3..df35d4ec46f0 100644 --- a/security/selinux/Makefile +++ b/security/selinux/Makefile @@ -26,5 +26,5 @@ quiet_cmd_flask = GEN $(obj)/flask.h $(obj)/av_permissions .h cmd_flask = $< $(obj)/flask.h $(obj)/av_permissions.h
targets += flask.h av_permissions.h -$(obj)/flask.h $(obj)/av_permissions.h &: scripts/selinux/genheaders/genheaders FORCE +$(obj)/flask.h $(obj)/av_permissions.h: scripts/selinux/genheaders/genheaders F ORCE $(call if_changed,flask)
On 2023-06-01 10:27, Paul Moore wrote:
On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino luizcap@amazon.com wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
...
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
Thanks Luiz, this is a fun one :/
It was a fun to debug TBH :-)
Based on a quick search, it looks like the grouped target may be the cause, especially for older (pre-4.3) versions of make. Looking through the rest of the kernel I don't see any other grouped targets, and in fact the top level Makefile even mentions holding off on using grouped targets until make v4.3 is common/required.
Exactly.
I don't have an older userspace immediately available, would you mind trying the fix/patch below to see if it resolves the problem on your system? It's a cut-n-paste so the patch may not apply directly, but it basically just removes the '&' from the make rule, turning it into an old-fashioned non-grouped target.
I tried the attached patch on top of latest Linus tree (ac2263b588dffd), but unfortunately I got the same issue which is puzzling. Reverting 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
I have no problem trying patches or helping debug, but if you want to give it a try on reproducing you could try with make-3.82 from:
https://ftp.gnu.org/gnu/make/make-3.82.tar.bz2
- Luiz
On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino luizcap@amazon.com wrote:
On 2023-06-01 10:27, Paul Moore wrote:
On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino luizcap@amazon.com wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
...
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
Thanks Luiz, this is a fun one :/
It was a fun to debug TBH :-)
Based on a quick search, it looks like the grouped target may be the cause, especially for older (pre-4.3) versions of make. Looking through the rest of the kernel I don't see any other grouped targets, and in fact the top level Makefile even mentions holding off on using grouped targets until make v4.3 is common/required.
Exactly.
I don't have an older userspace immediately available, would you mind trying the fix/patch below to see if it resolves the problem on your system? It's a cut-n-paste so the patch may not apply directly, but it basically just removes the '&' from the make rule, turning it into an old-fashioned non-grouped target.
I tried the attached patch on top of latest Linus tree (ac2263b588dffd), but unfortunately I got the same issue which is puzzling. Reverting 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
I'm at a bit of a loss here ... the only thing that seems to jump out is that the genheaders tool is run twice without the grouped target approach, but with both runs happening at the same point in the build and the second run updating both header files, I'm a bit at a loss as to why this would be problematic.
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
On 2023-06-01 11:45, Paul Moore wrote:
On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino luizcap@amazon.com wrote:
On 2023-06-01 10:27, Paul Moore wrote:
On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino luizcap@amazon.com wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
...
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
Thanks Luiz, this is a fun one :/
It was a fun to debug TBH :-)
Based on a quick search, it looks like the grouped target may be the cause, especially for older (pre-4.3) versions of make. Looking through the rest of the kernel I don't see any other grouped targets, and in fact the top level Makefile even mentions holding off on using grouped targets until make v4.3 is common/required.
Exactly.
I don't have an older userspace immediately available, would you mind trying the fix/patch below to see if it resolves the problem on your system? It's a cut-n-paste so the patch may not apply directly, but it basically just removes the '&' from the make rule, turning it into an old-fashioned non-grouped target.
I tried the attached patch on top of latest Linus tree (ac2263b588dffd), but unfortunately I got the same issue which is puzzling. Reverting 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
I'm at a bit of a loss here ... the only thing that seems to jump out is that the genheaders tool is run twice without the grouped target approach, but with both runs happening at the same point in the build and the second run updating both header files, I'm a bit at a loss as to why this would be problematic.
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
Thank you Paul, I really appreciate your fast response. I'd also appreciate if you CC me in the revert patch so that I don't loose track of it.
- Luiz
On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
On 2023-06-01 11:45, Paul Moore wrote:
On Thu, Jun 1, 2023 at 11:03 AM Luiz Capitulino luizcap@amazon.com wrote:
On 2023-06-01 10:27, Paul Moore wrote:
On Wed, May 31, 2023 at 10:13 PM Luiz Capitulino luizcap@amazon.com wrote:
Hi Paul,
A number of stable kernels recently backported this upstream commit:
""" commit 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 Author: Paul Moore paul@paul-moore.com Date: Wed Apr 12 13:29:11 2023 -0400
selinux: ensure av_permissions.h is built when needed"""
We're seeing a build issue with this commit where the "crash" tool will fail to start, it complains that the vmlinux image and /proc/version don't match.
A minimum reproducer would be having "make" version before 4.3 and building the kernel with:
$ make bzImages $ make modules
...
This only happens with commit 4ce1f694eb5 applied and older "make", in my case I have "make" version 3.82.
If I revert 4ce1f694eb5 or use "make" version 4.3 I get identical strings (except for the "Linux version" part):
Thanks Luiz, this is a fun one :/
It was a fun to debug TBH :-)
Based on a quick search, it looks like the grouped target may be the cause, especially for older (pre-4.3) versions of make. Looking through the rest of the kernel I don't see any other grouped targets, and in fact the top level Makefile even mentions holding off on using grouped targets until make v4.3 is common/required.
Exactly.
I don't have an older userspace immediately available, would you mind trying the fix/patch below to see if it resolves the problem on your system? It's a cut-n-paste so the patch may not apply directly, but it basically just removes the '&' from the make rule, turning it into an old-fashioned non-grouped target.
I tried the attached patch on top of latest Linus tree (ac2263b588dffd), but unfortunately I got the same issue which is puzzling. Reverting 4ce1f694eb5d8ca607fed8542d32a33b4f1217a5 does solve the issue though.
I'm at a bit of a loss here ... the only thing that seems to jump out is that the genheaders tool is run twice without the grouped target approach, but with both runs happening at the same point in the build and the second run updating both header files, I'm a bit at a loss as to why this would be problematic.
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
Thank you Paul, I really appreciate your fast response. I'd also appreciate if you CC me in the revert patch so that I don't loose track of it.
And please add a cc: stable@ to it too :)
On Thu, Jun 1, 2023 at 1:05 PM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
On 2023-06-01 11:45, Paul Moore wrote:
...
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
Thank you Paul, I really appreciate your fast response. I'd also appreciate if you CC me in the revert patch so that I don't loose track of it.
And please add a cc: stable@ to it too :)
Done, and done. I just sent the patch so give it a few minutes if you haven't seen it yet. I'll leave it for a few hours in case anyone can give it a test build before I send it up to Linus.
On 2023-06-01 14:10, Paul Moore wrote:
On Thu, Jun 1, 2023 at 1:05 PM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
On 2023-06-01 11:45, Paul Moore wrote:
...
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
Thank you Paul, I really appreciate your fast response. I'd also appreciate if you CC me in the revert patch so that I don't loose track of it.
And please add a cc: stable@ to it too :)
Done, and done. I just sent the patch so give it a few minutes if you haven't seen it yet. I'll leave it for a few hours in case anyone can give it a test build before I send it up to Linus.
Thanks Paul, I can test it but it may take more than a few hours for me.
- Luiz
On Thu, Jun 1, 2023 at 2:15 PM Luiz Capitulino luizcap@amazon.com wrote:
On 2023-06-01 14:10, Paul Moore wrote:
On Thu, Jun 1, 2023 at 1:05 PM Greg KH gregkh@linuxfoundation.org wrote:
On Thu, Jun 01, 2023 at 11:50:16AM -0400, Luiz Capitulino wrote:
On 2023-06-01 11:45, Paul Moore wrote:
...
I don't want to block on fixing the kernel build while I keep chasing some esoteric build behavior so I'm just going to revert the patch with a note to revisit this when we require make >= 4.3.
Regardless, thanks for the report and the help testing, expect a patch/revert shortly ...
Thank you Paul, I really appreciate your fast response. I'd also appreciate if you CC me in the revert patch so that I don't loose track of it.
And please add a cc: stable@ to it too :)
Done, and done. I just sent the patch so give it a few minutes if you haven't seen it yet. I'll leave it for a few hours in case anyone can give it a test build before I send it up to Linus.
Thanks Paul, I can test it but it may take more than a few hours for me.
Okay, no worries, I'll hold off until I hear back from you. I'd rather it take a little longer in order to get the warm fuzzies you get from a 'Tested-by:' by one of the 'Reported-by:' people :)
Thanks for all your help with this Luiz.
linux-stable-mirror@lists.linaro.org
-
Greg KH -
Luiz Capitulino -
Paul Moore