Hi stable tree maintainers,
Please revert the backports of
44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
namely:
5.4: 03475167fda50b8511ef620a27409b08365882e1 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93 6.1: b0cde867b80a5e81fcbc0383e138f5845f2005ee 6.6: 1a45994fb218d93dec48a3a86f68283db61e0936
There seems to be a bad interaction between this change and older PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation continues to see if this will need to be reverted from 6.6, 6.11, and mainline. But for now, it's clearly a problem for older kernels with older qemu.
Thanks!
-Kees
[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
21.11.2024 17:33, Kees Cook wrote:
Hi stable tree maintainers,
Please revert the backports of
44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
namely:
5.4: 03475167fda50b8511ef620a27409b08365882e1 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93 6.1: b0cde867b80a5e81fcbc0383e138f5845f2005ee 6.6: 1a45994fb218d93dec48a3a86f68283db61e0936
There seems to be a bad interaction between this change and older PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation continues to see if this will need to be reverted from 6.6, 6.11, and mainline. But for now, it's clearly a problem for older kernels with older qemu.
Thanks!
-Kees
[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
Unfortunately I haven't seen this thread and this email before now, when things are already too late.
And it turned out it's entirely my fault with all this. Let me explain so things become clear to everyone.
The problem here is entirely in qemu-user. The fundamental issue is that qemu-user does not implement an MMU, instead, it implements just address shift, searching for a memory region for the guest address space which is hopefully not used by qemu-user itself.
In practice, this is rarely an issue though, when - and this is the default - qemu is built as a static-pie executable. This is important: it's the default mode for the static build - it builds as static-pie executable, which works around the problem in almost all cases. This is done for quite a long time, too.
However, I, as qemu maintainer in debian, got a bug report saying that qemu-user-static isn't "static enough" - because for some tools used on debian (lintian), static-pie was something unknown and the tool issued a warning. And at the time, I just added --disable-pie flag to the build, without much thinking. This is where things went wrong.
Later I reverted this change with a shame, because it causes numerous configurations to fail randomly, and each of them is very difficult to debug (especially due to randomness of failures, sometimes it can work 50 times in a row but fail on the 51th).
But unfortunately, I forgot to revert this "de-PIEsation" change in debian stable, and that's exactly where the original bug report come from, stating kernel broke builds in qemu.
The same qemu-user-static configuration has been used by some other distributions too, but hopefully everything's fixed now. Except of debian bookworm, and probably also ubuntu jammy (previous LTS).
It is not an "older qemu" anymore (though for a very old qemu this is true again, that old one can't be used anymore with modern executables anyway due to other reasons). It is just my build mistake which is *still* unfixed on debian stable (bookworm). And even there, this issue can trivially be fixed locally, since qemu-user-static is self-contained and can be installed on older debian releases, and I always provide up-to-date backports of qemu packages for debian stable.
And yes, qemu had numerous improvements in this area since bookworm version, which addressed many other issues around this and fixed many other configurations (which are not related to this kernel change), but the fundamental issue (lack of full-blown MMU) remains.
Hopefully this clears things up, and it can be seen that this is not a kernel bug. And I'm hoping we'll fix this in debian bookworm soon too.
Thanks, and sorry for all the buzz which caused my 2 mistakes.
/mjt
Hi all,
On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
21.11.2024 17:33, Kees Cook wrote:
Hi stable tree maintainers,
Please revert the backports of
44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
namely:
5.4: 03475167fda50b8511ef620a27409b08365882e1 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93 6.1: b0cde867b80a5e81fcbc0383e138f5845f2005ee 6.6: 1a45994fb218d93dec48a3a86f68283db61e0936
There seems to be a bad interaction between this change and older PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation continues to see if this will need to be reverted from 6.6, 6.11, and mainline. But for now, it's clearly a problem for older kernels with older qemu.
Thanks!
-Kees
[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
Unfortunately I haven't seen this thread and this email before now, when things are already too late.
And it turned out it's entirely my fault with all this. Let me explain so things become clear to everyone.
The problem here is entirely in qemu-user. The fundamental issue is that qemu-user does not implement an MMU, instead, it implements just address shift, searching for a memory region for the guest address space which is hopefully not used by qemu-user itself.
In practice, this is rarely an issue though, when - and this is the default - qemu is built as a static-pie executable. This is important: it's the default mode for the static build - it builds as static-pie executable, which works around the problem in almost all cases. This is done for quite a long time, too.
However, I, as qemu maintainer in debian, got a bug report saying that qemu-user-static isn't "static enough" - because for some tools used on debian (lintian), static-pie was something unknown and the tool issued a warning. And at the time, I just added --disable-pie flag to the build, without much thinking. This is where things went wrong.
Later I reverted this change with a shame, because it causes numerous configurations to fail randomly, and each of them is very difficult to debug (especially due to randomness of failures, sometimes it can work 50 times in a row but fail on the 51th).
But unfortunately, I forgot to revert this "de-PIEsation" change in debian stable, and that's exactly where the original bug report come from, stating kernel broke builds in qemu.
The same qemu-user-static configuration has been used by some other distributions too, but hopefully everything's fixed now. Except of debian bookworm, and probably also ubuntu jammy (previous LTS).
It is not an "older qemu" anymore (though for a very old qemu this is true again, that old one can't be used anymore with modern executables anyway due to other reasons). It is just my build mistake which is *still* unfixed on debian stable (bookworm). And even there, this issue can trivially be fixed locally, since qemu-user-static is self-contained and can be installed on older debian releases, and I always provide up-to-date backports of qemu packages for debian stable.
And yes, qemu had numerous improvements in this area since bookworm version, which addressed many other issues around this and fixed many other configurations (which are not related to this kernel change), but the fundamental issue (lack of full-blown MMU) remains.
Hopefully this clears things up, and it can be seen that this is not a kernel bug. And I'm hoping we'll fix this in debian bookworm soon too.
Thanks, and sorry for all the buzz which caused my 2 mistakes.
So catching up with that as we currently did cherry-pick the revert in Debian but I defintivelfy would like to align with upstream (and drop the cherry-pick again if it's not going to be picked for 6.1.y upstream):
I'm a bit lost here. What are we going to do? Is the commit still temporarly be applied to the stable series or are we staying at the status quo and we should solely deal it within Debian on qemu side to address the issue above and then we are fine?
Or are there other cases outside Debian making it necessary apply the above proposed revert to the stable series?
Regards, Salvatore
On Thu, Dec 05, 2024 at 07:54:40AM +0100, Salvatore Bonaccorso wrote:
Hi all,
On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
21.11.2024 17:33, Kees Cook wrote:
Hi stable tree maintainers,
Please revert the backports of
44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
namely:
5.4: 03475167fda50b8511ef620a27409b08365882e1 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93 6.1: b0cde867b80a5e81fcbc0383e138f5845f2005ee 6.6: 1a45994fb218d93dec48a3a86f68283db61e0936
There seems to be a bad interaction between this change and older PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation continues to see if this will need to be reverted from 6.6, 6.11, and mainline. But for now, it's clearly a problem for older kernels with older qemu.
Thanks!
-Kees
[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
Unfortunately I haven't seen this thread and this email before now, when things are already too late.
And it turned out it's entirely my fault with all this. Let me explain so things become clear to everyone.
The problem here is entirely in qemu-user. The fundamental issue is that qemu-user does not implement an MMU, instead, it implements just address shift, searching for a memory region for the guest address space which is hopefully not used by qemu-user itself.
In practice, this is rarely an issue though, when - and this is the default - qemu is built as a static-pie executable. This is important: it's the default mode for the static build - it builds as static-pie executable, which works around the problem in almost all cases. This is done for quite a long time, too.
However, I, as qemu maintainer in debian, got a bug report saying that qemu-user-static isn't "static enough" - because for some tools used on debian (lintian), static-pie was something unknown and the tool issued a warning. And at the time, I just added --disable-pie flag to the build, without much thinking. This is where things went wrong.
Later I reverted this change with a shame, because it causes numerous configurations to fail randomly, and each of them is very difficult to debug (especially due to randomness of failures, sometimes it can work 50 times in a row but fail on the 51th).
But unfortunately, I forgot to revert this "de-PIEsation" change in debian stable, and that's exactly where the original bug report come from, stating kernel broke builds in qemu.
The same qemu-user-static configuration has been used by some other distributions too, but hopefully everything's fixed now. Except of debian bookworm, and probably also ubuntu jammy (previous LTS).
It is not an "older qemu" anymore (though for a very old qemu this is true again, that old one can't be used anymore with modern executables anyway due to other reasons). It is just my build mistake which is *still* unfixed on debian stable (bookworm). And even there, this issue can trivially be fixed locally, since qemu-user-static is self-contained and can be installed on older debian releases, and I always provide up-to-date backports of qemu packages for debian stable.
And yes, qemu had numerous improvements in this area since bookworm version, which addressed many other issues around this and fixed many other configurations (which are not related to this kernel change), but the fundamental issue (lack of full-blown MMU) remains.
Hopefully this clears things up, and it can be seen that this is not a kernel bug. And I'm hoping we'll fix this in debian bookworm soon too.
Thanks, and sorry for all the buzz which caused my 2 mistakes.
So catching up with that as we currently did cherry-pick the revert in Debian but I defintivelfy would like to align with upstream (and drop the cherry-pick again if it's not going to be picked for 6.1.y upstream):
I'm a bit lost here. What are we going to do? Is the commit still temporarly be applied to the stable series or are we staying at the status quo and we should solely deal it within Debian on qemu side to address the issue above and then we are fine?
I read this as "oops, we messed up in qemu and will fix it there" and so I dropped the reverts from the kernel. If that's not the case, please let me know.
thanks,
greg k-h
Hi Greg,
On Thu, Dec 05, 2024 at 08:07:15AM +0100, Greg KH wrote:
On Thu, Dec 05, 2024 at 07:54:40AM +0100, Salvatore Bonaccorso wrote:
Hi all,
On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
21.11.2024 17:33, Kees Cook wrote:
Hi stable tree maintainers,
Please revert the backports of
44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
namely:
5.4: 03475167fda50b8511ef620a27409b08365882e1 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93 6.1: b0cde867b80a5e81fcbc0383e138f5845f2005ee 6.6: 1a45994fb218d93dec48a3a86f68283db61e0936
There seems to be a bad interaction between this change and older PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation continues to see if this will need to be reverted from 6.6, 6.11, and mainline. But for now, it's clearly a problem for older kernels with older qemu.
Thanks!
-Kees
[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
Unfortunately I haven't seen this thread and this email before now, when things are already too late.
And it turned out it's entirely my fault with all this. Let me explain so things become clear to everyone.
The problem here is entirely in qemu-user. The fundamental issue is that qemu-user does not implement an MMU, instead, it implements just address shift, searching for a memory region for the guest address space which is hopefully not used by qemu-user itself.
In practice, this is rarely an issue though, when - and this is the default - qemu is built as a static-pie executable. This is important: it's the default mode for the static build - it builds as static-pie executable, which works around the problem in almost all cases. This is done for quite a long time, too.
However, I, as qemu maintainer in debian, got a bug report saying that qemu-user-static isn't "static enough" - because for some tools used on debian (lintian), static-pie was something unknown and the tool issued a warning. And at the time, I just added --disable-pie flag to the build, without much thinking. This is where things went wrong.
Later I reverted this change with a shame, because it causes numerous configurations to fail randomly, and each of them is very difficult to debug (especially due to randomness of failures, sometimes it can work 50 times in a row but fail on the 51th).
But unfortunately, I forgot to revert this "de-PIEsation" change in debian stable, and that's exactly where the original bug report come from, stating kernel broke builds in qemu.
The same qemu-user-static configuration has been used by some other distributions too, but hopefully everything's fixed now. Except of debian bookworm, and probably also ubuntu jammy (previous LTS).
It is not an "older qemu" anymore (though for a very old qemu this is true again, that old one can't be used anymore with modern executables anyway due to other reasons). It is just my build mistake which is *still* unfixed on debian stable (bookworm). And even there, this issue can trivially be fixed locally, since qemu-user-static is self-contained and can be installed on older debian releases, and I always provide up-to-date backports of qemu packages for debian stable.
And yes, qemu had numerous improvements in this area since bookworm version, which addressed many other issues around this and fixed many other configurations (which are not related to this kernel change), but the fundamental issue (lack of full-blown MMU) remains.
Hopefully this clears things up, and it can be seen that this is not a kernel bug. And I'm hoping we'll fix this in debian bookworm soon too.
Thanks, and sorry for all the buzz which caused my 2 mistakes.
So catching up with that as we currently did cherry-pick the revert in Debian but I defintivelfy would like to align with upstream (and drop the cherry-pick again if it's not going to be picked for 6.1.y upstream):
I'm a bit lost here. What are we going to do? Is the commit still temporarly be applied to the stable series or are we staying at the status quo and we should solely deal it within Debian on qemu side to address the issue above and then we are fine?
I read this as "oops, we messed up in qemu and will fix it there" and so I dropped the reverts from the kernel. If that's not the case, please let me know.
Thanks for the quick confirmation.
Michael, I will drop the local revert in src:linux then on the next upload I do. Can you handle the qemu part in Debian accordingly (and make sure the fix lands in the next Debian point release in January and ideally maybe via stable-updates earlier to affected people?)
Regards, Salvatore
05.12.2024 10:22, Salvatore Bonaccorso wrote:
Michael, I will drop the local revert in src:linux then on the next upload I do. Can you handle the qemu part in Debian accordingly (and make sure the fix lands in the next Debian point release in January and ideally maybe via stable-updates earlier to affected people?)
It would be great to fix this in debian, yes. It's a debian matter anyway, for details there's no need to bother kernel people. Please see #1088273. Fixed qemu release should come together with the updated kernel in debian -- if it will be next point release, so be it, but if the kernel goes before, we should pick qemu too, because else we'll re-introduce #1087822.
Ubuntu is also affected, it has similar situation, but they have different schedule and different versions.
Thanks,
/mjt
Hi mjt,
Michael Tokarev wrote on Thu, Dec 05, 2024 at 10:35:26AM +0300:
It would be great to fix this in debian, yes. It's a debian matter anyway, for details there's no need to bother kernel people. Please see #1088273. Fixed qemu release should come together with the updated kernel in debian -- if it will be next point release, so be it, but if the kernel goes before, we should pick qemu too, because else we'll re-introduce #1087822.
The kernel aleady shipped in debian a while ago, hence the bug reports :)
More precisely, the "bad" commit b0cde867b80 (in 6.1 tree) made it in 6.1.107, and debian stable has been shipping it since the 6.1.112 update on the 3rd of October[1][2], so we've been here for a while now
[1] https://tracker.debian.org/pkg/linux-signed-amd64 [2] https://tracker.debian.org/news/1571617/accepted-linux-signed-amd64-611121-s...
Cheers,
linux-stable-mirror@lists.linaro.org
-
Dominique Martinet -
Greg KH -
Kees Cook -
Michael Tokarev -
Salvatore Bonaccorso