The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org --- Changes in v2: - Expand the scope to arch/arm64 instead of just physaddr.c - Link to v1: https://lore.kernel.org/all/20251231-annotated-v1-1-9db1c0d03062@debian.org/ --- arch/arm64/Kbuild | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild index 5bfbf7d79c99..d876bc0e5421 100644 --- a/arch/arm64/Kbuild +++ b/arch/arm64/Kbuild @@ -1,4 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-only + +# Branch profiling isn't noinstr-safe +subdir-ccflags-$(CONFIG_TRACE_BRANCH_PROFILING) += -DDISABLE_BRANCH_PROFILING + obj-y += kernel/ mm/ net/ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/
--- base-commit: c8ebd433459bcbf068682b09544e830acd7ed222 change-id: 20251231-annotated-75de3f33cd7b
Best regards, -- Breno Leitao leitao@debian.org
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL is unsound in a number of places, so I'd prefer to remove that Fixes tag and backport this for all stable trees.
Regardless, I'm in favour of disabling CONFIG_DEBUG_VIRTUAL widely, so:
Acked-by: Mark Rutland mark.rutland@arm.com
Mark.
Changes in v2:
- Expand the scope to arch/arm64 instead of just physaddr.c
- Link to v1: https://lore.kernel.org/all/20251231-annotated-v1-1-9db1c0d03062@debian.org/
arch/arm64/Kbuild | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild index 5bfbf7d79c99..d876bc0e5421 100644 --- a/arch/arm64/Kbuild +++ b/arch/arm64/Kbuild @@ -1,4 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-only
+# Branch profiling isn't noinstr-safe +subdir-ccflags-$(CONFIG_TRACE_BRANCH_PROFILING) += -DDISABLE_BRANCH_PROFILING
obj-y += kernel/ mm/ net/ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/
base-commit: c8ebd433459bcbf068682b09544e830acd7ed222 change-id: 20251231-annotated-75de3f33cd7b
Best regards,
Breno Leitao leitao@debian.org
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL is unsound in a number of places, so I'd prefer to remove that Fixes tag and backport this for all stable trees.
Regardless, I'm in favour of disabling CONFIG_DEBUG_VIRTUAL widely, so:
Acked-by: Mark Rutland mark.rutland@arm.com
Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both places in my reply.
Mark.
On Tue, 6 Jan 2026 12:24:09 +0000 Mark Rutland mark.rutland@arm.com wrote:
Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both places in my reply.
Note, it could still be useful ;-)
I've been running my yearly branch profiling build on both my main workstation and my server. I post the results from my server publicly (this is updated every night):
https://rostedt.org/branches/current
If you check out the branch_annotated file, you can see there's still quite a bit that gets it wrong. Some of these is because of bad assumptions by the developer, others is because the code moved around causing new branches to make later annotated branches go the opposite way.
Note, its still slow to download, not only because my internet is 20mbs up, but also because that web server is running on a machine with this profiling still enabled. I'll rebuild the kernel and disable it this weekend.
-- Steve
Hello Steven,
On Tue, Jan 06, 2026 at 11:11:42AM -0500, Steven Rostedt wrote:
On Tue, 6 Jan 2026 12:24:09 +0000 Mark Rutland mark.rutland@arm.com wrote:
Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both places in my reply.
Note, it could still be useful ;-)
I've been running my yearly branch profiling build on both my main workstation and my server. I post the results from my server publicly (this is updated every night):
https://rostedt.org/branches/current
If you check out the branch_annotated file, you can see there's still quite a bit that gets it wrong. Some of these is because of bad assumptions by the developer, others is because the code moved around causing new branches to make later annotated branches go the opposite way.
I am starting to look and remove some of these likely/unlikely hint that are 100% wrong on some very sane configuration (arm64 baremetal hosts running a webserver).
So far, these are the fixes I have in flight now.
* https://lore.kernel.org/all/20260105-dcache-v1-1-f0d904b4a7c2@debian.org/ * https://lore.kernel.org/all/20260106-blk_unlikely-v1-1-90fb556a6776@debian.o... * https://lore.kernel.org/all/20260105-annotated_idle-v1-1-10ddf0771b58@debian...
On Wed, 7 Jan 2026 01:37:36 -0800 Breno Leitao leitao@debian.org wrote:
I am starting to look and remove some of these likely/unlikely hint that are 100% wrong on some very sane configuration (arm64 baremetal hosts running a webserver).
So far, these are the fixes I have in flight now.
Nice. Thanks for letting me know. I'm happy to see that someone besides me is using the annotated branch profiler to fix real issues in the kernel.
I would love it if more people did the same. Anyway, please advertise this as much as possible. If it shows that it is helping to fix real bugs, maybe it would attract the interest of more developers, and more importantly, it keeps the tooling in place. I'm constantly fighting to keep it from being removed from the kernel :-p
-- Steve
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL is unsound in a number of places, so I'd prefer to remove that Fixes tag and backport this for all stable trees.
That is fair, thanks for the review.
Should I submit a new version without the fixes tag, or, do you guys do it while merging the patch?
thanks --breno
On Tue, Jan 06, 2026 at 06:05:37AM -0800, Breno Leitao wrote:
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL is unsound in a number of places, so I'd prefer to remove that Fixes tag and backport this for all stable trees.
That is fair, thanks for the review.
Should I submit a new version without the fixes tag, or, do you guys do it while merging the patch?
I assume that Catalin or Will can handle that when applying (if they agree with me); no need to respin.
Mark.
On Tue, Jan 06, 2026 at 03:31:16PM +0000, Mark Rutland wrote:
On Tue, Jan 06, 2026 at 06:05:37AM -0800, Breno Leitao wrote:
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch profiling for the entire arch/arm64 tree, similar to what x86 already does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL") Signed-off-by: Breno Leitao leitao@debian.org
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL is unsound in a number of places, so I'd prefer to remove that Fixes tag and backport this for all stable trees.
That is fair, thanks for the review.
Should I submit a new version without the fixes tag, or, do you guys do it while merging the patch?
I assume that Catalin or Will can handle that when applying (if they agree with me); no need to respin.
Thanks Mark.
On Tue, 06 Jan 2026 02:16:35 -0800, Breno Leitao wrote:
The arm64 kernel doesn't boot with annotated branches (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm solved the problem. Narrowing down a bit further, I found that physaddr.c is the file that needs to have branch profiling disabled to get the machine to boot.
[...]
Applied to arm64 (for-next/misc), thanks!
[1/1] arm64: Disable branch profiling for all arm64 code https://git.kernel.org/arm64/c/f22c81bebf8b
Cheers,
linux-stable-mirror@lists.linaro.org
-
Breno Leitao -
Mark Rutland -
Steven Rostedt -
Will Deacon