Hi Greg, Our internal rpm build for aarch64 started failing after merging v5.15.60. The build actually succeeds, but packaging failed due to tools which extract the Build ID from vmlinux.
Expected: readelf -n vmlinux
Displaying notes found in: .notes Owner Data size Description GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: ae5803ded69a15fb0a75905ec226a76cc1823d22 Linux 0x00000004 func description data: 00 00 00 00 Linux 0x00000001 OPEN description data: 00
Actual (post v5.15.60 merge): readelf -n vmlinux <no output>
Digging deeper...
Expected: readelf --headers vmlinux | sed -ne '/Program Header/,$p' Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x0000000000010000 0xffff800008000000 0xffff800008000000 0x0000000002012a00 0x000000000208f724 RWE 0x10000 NOTE 0x0000000001705948 0xffff8000096f5948 0xffff8000096f5948 0x0000000000000054 0x0000000000000054 R 0x4 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RW 0x10
Section to Segment mapping: Segment Sections... 00 .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss 01 .notes 02
Actual: readelf --headers vmlinux | sed -ne '/Program Header/,$p' Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x0000000000010000 0xffff800008000000 0xffff800008000000 0x0000000002012a00 0x000000000208f724 RWE 0x10000 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RW 0x10
Section to Segment mapping: Segment Sections... 00 .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss 01
readelf -n fails since NOTE segment is missing from vmlinux.
Why?
5.15 Commit: 4c7ee827da2c ("Makefile: link with -z noexecstack --no-warn-rwx-segments") surfaced this issue.
1. Reverting this patch fixed this issue.
2. Removing all .note.GNU-stack sections in cmd_modversions_S also worked:
diff --git a/scripts/Makefile.build b/scripts/Makefile.build index 17aa8ef2d52a..b95cfbb43cee 100644 --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -379,6 +379,8 @@ cmd_modversions_S = \ if $(OBJDUMP) -h $@ | grep -q __ksymtab; then \ $(call cmd_gensymtypes_S,$(KBUILD_SYMTYPES),$(@:.o=.symtypes)) \ > $(@D)/.tmp_$(@F:.o=.ver); \ + echo "SECTIONS { /DISCARD/ : { *(.note.GNU-stack) }}" \ + >> $(@D)/.tmp_$(@F:.o=.ver); \ \ $(LD) $(KBUILD_LDFLAGS) -r -o $(@D)/.tmp_$(@F) $@ \ -T $(@D)/.tmp_$(@F:.o=.ver); \
3. But the simplest fix I found was to remove -z noexecstack just for head.S (this patch).
This issue exists, in my testing of arm64 builds, for stable versions 5.15.30+, 5.10.136+, and 5.4.210+ and reproduces with every attempted combination of gcc{11,12} ld{2.36,2.37,2.38,2.39}.
It can also be reproduced upstream by cherry-picking 0d362be5b142 on-top until 7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS") which was part of merge df202b452fe6 ("Merge tag 'kbuild-v5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild")
It seems kernels prior to v5.4 and 5.19+ use differing KBUILD rules (with CONFIG_MODVERSIONS=y) for linking components of vmlinux, and in my testing these other versions did not encounter this issue.
Other architectures besides arm64 might also have similar bug/feature of ld. Though, x86_64 v5.15.60 worked as expected, so no further experiments were performed for x86_64.
arm64 repro test:
make ARCH=arm64 mrproper cp arch/arm64/configs/defconfig ".config" scripts/config -e MODVERSIONS make ARCH=arm64 olddefconfig make ARCH=arm64 -j16 vmlinux readelf -n vmlinux | grep -F "Build ID:"
Please consider applying for 5.15, 5.10, and 5.4.
Regards,
--Tom
Tom Saeger (1): arm64: fix Build ID if CONFIG_MODVERSIONS
arch/arm64/kernel/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
base-commit: e4a7232c917cd1b56d5b4fa9d7a23e3eabfecba0
Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions. This kernel's KBUILD tooling uses both relocatable (-r) and (-z noexecstack) to link head.o which results in ld adding a .note.GNU-stack section. Final linking of vmlinux should add a .NOTES segment containing the Build ID, but does NOT if head.o has a .note.GNU-stack section.
Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to prevent .note.GNU-stack from being added to head.o. Final link of vmlinux then properly adds .NOTES segment containing Build ID that can be read using tools like 'readelf -n'.
Cc: stable@vger.kernel.org # 5.15, 5.10, 5.4 Signed-off-by: Tom Saeger tom.saeger@oracle.com --- arch/arm64/kernel/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 749e31475e41..8bdee0f655e2 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -85,3 +85,8 @@ extra-y += $(head-y) vmlinux.lds ifeq ($(CONFIG_DEBUG_EFI),y) AFLAGS_head.o += -DVMLINUX_PATH=""$(realpath $(objtree)/vmlinux)"" endif + +ifeq ($(CONFIG_MODVERSIONS),y) +# filter-out -z noexecstack for head.S +$(obj)/head.o: KBUILD_LDFLAGS := $(shell echo "$(KBUILD_LDFLAGS)" | sed -e 's/-z\s+noexecstack//') +endif
On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions. This kernel's KBUILD tooling uses both relocatable (-r) and (-z noexecstack) to link head.o which results in ld adding a .note.GNU-stack section. Final linking of vmlinux should add a .NOTES segment containing the Build ID, but does NOT if head.o has a .note.GNU-stack section.
Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to prevent .note.GNU-stack from being added to head.o. Final link of vmlinux then properly adds .NOTES segment containing Build ID that can be read using tools like 'readelf -n'.
Cc: stable@vger.kernel.org # 5.15, 5.10, 5.4 Signed-off-by: Tom Saeger tom.saeger@oracle.com
arch/arm64/kernel/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Why isn't this needed in Linus's tree?
And why not cc: everyone involved in this, I would need acks from maintainers to be able to accept this...
thanks,
greg k-h
On Thu, Dec 08, 2022 at 09:34:40PM +0100, Greg Kroah-Hartman wrote:
On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions. This kernel's KBUILD tooling uses both relocatable (-r) and (-z noexecstack) to link head.o which results in ld adding a .note.GNU-stack section. Final linking of vmlinux should add a .NOTES segment containing the Build ID, but does NOT if head.o has a .note.GNU-stack section.
Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to prevent .note.GNU-stack from being added to head.o. Final link of vmlinux then properly adds .NOTES segment containing Build ID that can be read using tools like 'readelf -n'.
Cc: stable@vger.kernel.org # 5.15, 5.10, 5.4 Signed-off-by: Tom Saeger tom.saeger@oracle.com
arch/arm64/kernel/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Why isn't this needed in Linus's tree?
0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
was merged after 7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")
Linus's tree never had -z noexecstack with these same KBUILD rules.
And why not cc: everyone involved in this, I would need acks from maintainers to be able to accept this...
Fair request.
Between ~5.3 and 5.19-rc1 cherry-picking 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") and building arm64 with CONFIG_MODVERSIONS=y results in vmlinux missing Build ID
head.S is compiled to head.o head.o is linked (ld) with -r and -z noexecstack which adds .note.GNU-stack section in head.o head.o is then linked again with vmlinux (resulting vmlinux is missing .NOTE segment)
Can folks confirm/deny ld behavior is expected (arm64)? And whether the above patch would be an acceptable fix for these kernel versions?
repro test in cover letter: https://lore.kernel.org/all/cover.1670358255.git.tom.saeger@oracle.com/#r
Regards, --Tom
thanks,
greg k-h
On Thu, Dec 08, 2022 at 05:31:06PM -0600, Tom Saeger wrote:
On Thu, Dec 08, 2022 at 09:34:40PM +0100, Greg Kroah-Hartman wrote:
On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions. This kernel's KBUILD tooling uses both relocatable (-r) and (-z noexecstack) to link head.o which results in ld adding a .note.GNU-stack section. Final linking of vmlinux should add a .NOTES segment containing the Build ID, but does NOT if head.o has a .note.GNU-stack section.
Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to prevent .note.GNU-stack from being added to head.o. Final link of vmlinux then properly adds .NOTES segment containing Build ID that can be read using tools like 'readelf -n'.
Cc: stable@vger.kernel.org # 5.15, 5.10, 5.4 Signed-off-by: Tom Saeger tom.saeger@oracle.com
arch/arm64/kernel/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Why isn't this needed in Linus's tree?
0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
was merged after 7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")
Linus's tree never had -z noexecstack with these same KBUILD rules.
Then it needs to say that, in detail, in this changelog please.
And why not cc: everyone involved in this, I would need acks from maintainers to be able to accept this...
Fair request.
Between ~5.3 and 5.19-rc1 cherry-picking 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") and building arm64 with CONFIG_MODVERSIONS=y results in vmlinux missing Build ID
head.S is compiled to head.o head.o is linked (ld) with -r and -z noexecstack which adds .note.GNU-stack section in head.o head.o is then linked again with vmlinux (resulting vmlinux is missing .NOTE segment)
Can folks confirm/deny ld behavior is expected (arm64)? And whether the above patch would be an acceptable fix for these kernel versions?
repro test in cover letter: https://lore.kernel.org/all/cover.1670358255.git.tom.saeger@oracle.com/#r
Please resend, no need for a cover letter for just one patch, and include all of the needed info in the changelog itself.
thanks,
greg k-h
On Thu, Dec 8, 2022 at 3:31 PM Tom Saeger tom.saeger@oracle.com wrote:
Can folks confirm/deny ld behavior is expected (arm64)? And whether the above patch would be an acceptable fix for these kernel versions?
If you remove `-z noexecstack`, aren't you just going to trigger warnings from BFD again?
At the least consider adding a fixes tag for 0d362be5b142, and note that stable doesn't have 7b4537199a4a, in the commit message.
On Fri, Dec 09, 2022 at 09:58:42AM -0800, Nick Desaulniers wrote:
On Thu, Dec 8, 2022 at 3:31 PM Tom Saeger tom.saeger@oracle.com wrote:
Can folks confirm/deny ld behavior is expected (arm64)? And whether the above patch would be an acceptable fix for these kernel versions?
If you remove `-z noexecstack`, aren't you just going to trigger warnings from BFD again?
hmm, probably so, let me check
At the least consider adding a fixes tag for 0d362be5b142, and note that stable doesn't have 7b4537199a4a, in the commit message. -- Thanks, ~Nick Desaulniers
linux-stable-mirror@lists.linaro.org