We're currently trying to update the kernel in our distro (Flatcar) from v5.15.x to v6.1.x. When testing on Equinix Metal Ampere instances (c3.large.arm64) we now get a soft lockup about a minute after boot.
Has anyone else seen this? The splat looks like this, full dmesg is at [1] (trying without the attachement this time as LKML detects my mail as spam :/)
[ 84.297829] watchdog: BUG: soft lockup - CPU#45 stuck for 26s! [kworker/45:1:474] [ 84.297834] Modules linked in: veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink bonding mlx5_ib ipmi_ssif ipmi_devintf ib_core ipmi_msghandler evdev sch_fq_codel fuse configfs dmi_sysfs nls_ascii nls_cp437 vfat fat ext4 crc16 mbcache jbd2 dm_verity dm_bufio nvme nvme_core xhci_pci t10_pi xhci_hcd crc64_rocksoft_generic crc64_rocksoft igb crc_t10dif i2c_algo_bit crct10dif_generic mlx5_core usbcore i2c_core crc64 crct10dif_common usb_common hwmon pci_hyperv_intf btrfs blake2b_generic xor xor_neon lzo_compress zlib_deflate raid6_pq zstd_compress libcrc32c crc32c_generic dm_mirror dm_region_hash dm_log qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod scsi_mod scsi_common br_netfilter bridge stp llc overlay [ 84.435381] CPU: 45 PID: 474 Comm: kworker/45:1 Not tainted 6.1.30-flatcar #1 [ 84.447678] Hardware name: GIGABYTE R272-P30-JG/MP32-AR0-JG, BIOS F17a (SCP: 1.07.20210713) 07/22/2021 [ 84.467226] Workqueue: rcu_par_gp sync_rcu_exp_select_node_cpus [ 84.478239] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 84.490336] pc : smp_call_function_single+0xe4/0x1d0 [ 84.500287] lr : __sync_rcu_exp_select_node_cpus+0x278/0x480 [ 84.510869] sp : ffff80000b99bcc0 [ 84.518979] x29: ffff80000b99bcc0 x28: 0000000000000200 x27: ffffab5be3134080 [ 84.530897] x26: ffffab5be302d6a8 x25: ffff07ff89f01f40 x24: ffff083e5fba8540 [ 84.542751] x23: ffffab5be2e43ab0 x22: 0000000000000a00 x21: ffffab5be2e47540 [ 84.554510] x20: 000000000000f5ff x19: ffff80000b99bce0 x18: ffff80002be3bc58 [ 84.566213] x17: 0000000000000000 x16: ffffab5be06cdfc0 x15: 0000000000000001 [ 84.577841] x14: 0000000000000000 x13: 0000000000000010 x12: ffffab5be2e43ab0 [ 84.589361] x11: 0000000000000001 x10: 0000000000000040 x9 : ffffab5be312c270 [ 84.600832] x8 : ffffab5be2e34008 x7 : 0000000000028fb0 x6 : ffffab5bdfd0ef20 [ 84.612285] x5 : 0000000000000000 x4 : ffff083e5fc1c748 x3 : 0000000000000001 [ 84.623812] x2 : 0000000000000000 x1 : ffff083e5fc1c740 x0 : 0000000000000029 [ 84.635277] Call trace: [ 84.641994] smp_call_function_single+0xe4/0x1d0 [ 84.650957] __sync_rcu_exp_select_node_cpus+0x278/0x480 [ 84.660664] sync_rcu_exp_select_node_cpus+0x14/0x20 [ 84.669988] process_one_work+0x214/0x490 [ 84.678313] worker_thread+0x6c/0x430 [ 84.686206] kthread+0x108/0x10c [ 84.693668] ret_from_fork+0x10/0x20 [ 84.701417] Kernel panic - not syncing: softlockup: hung tasks [ 84.711437] CPU: 45 PID: 474 Comm: kworker/45:1 Tainted: G L 6.1.30-flatcar #1 [ 84.724261] Hardware name: GIGABYTE R272-P30-JG/MP32-AR0-JG, BIOS F17a (SCP: 1.07.20210713) 07/22/2021 [ 84.741870] Workqueue: rcu_par_gp sync_rcu_exp_select_node_cpus [ 84.751900] Call trace: [ 84.758325] dump_backtrace+0xe0/0x140 [ 84.765958] show_stack+0x18/0x30 [ 84.773057] dump_stack_lvl+0x64/0x80 [ 84.780384] dump_stack+0x18/0x34 [ 84.787220] panic+0x180/0x358 [ 84.793698] watchdog_nmi_enable+0x0/0x10 [ 84.801040] __hrtimer_run_queues+0x17c/0x340 [ 84.808624] hrtimer_interrupt+0xe8/0x244 [ 84.815807] arch_timer_handler_phys+0x2c/0x44 [ 84.823388] handle_percpu_devid_irq+0x88/0x230 [ 84.830995] generic_handle_domain_irq+0x2c/0x44 [ 84.838682] gic_handle_irq+0x50/0x140 [ 84.845468] call_on_irq_stack+0x24/0x4c [ 84.852402] do_interrupt_handler+0x80/0x84 [ 84.859569] el1_interrupt+0x34/0x6c [ 84.866100] el1h_64_irq_handler+0x18/0x2c [ 84.873134] el1h_64_irq+0x64/0x68 [ 84.879415] smp_call_function_single+0xe4/0x1d0 [ 84.886933] __sync_rcu_exp_select_node_cpus+0x278/0x480 [ 84.895205] sync_rcu_exp_select_node_cpus+0x14/0x20 [ 84.903118] process_one_work+0x214/0x490 [ 84.910040] worker_thread+0x6c/0x430 [ 84.916645] kthread+0x108/0x10c [ 84.922733] ret_from_fork+0x10/0x20
Thanks, Jeremi
[1]: https://gist.githubusercontent.com/dongsupark/3ab5fc464a995623a14542edd9b193...
On Tue, Jun 06, 2023 at 07:47:52AM -0700, Jeremi Piotrowski wrote:
We're currently trying to update the kernel in our distro (Flatcar) from v5.15.x to v6.1.x. When testing on Equinix Metal Ampere instances (c3.large.arm64) we now get a soft lockup about a minute after boot.
Has anyone else seen this? The splat looks like this, full dmesg is at [1] (trying without the attachement this time as LKML detects my mail as spam :/)
You might have better luck asking on the ARM lists, the stable@ list is for patches to add or other fixes, not generic failures like this one.
Also, why not take a working 6.1.x kernel for this platform (which I know they are out there) and comparing it to yours?
Good luck!
greg k-h
[ Upstream commit eb684408f3ea4856639675d6465f0024e498e4b1 ]
Instead of using the SMBIOS type 1 record 'family' field, which is often modified by OEMs, use the type 4 'processor ID' and 'processor version' fields, which are set to a small set of probe-able values on all known Ampere EFI systems in the field.
Fixes: 550b33cfd4452968 ("arm64: efi: Force the use of ...") Tested-by: Andrea Righi andrea.righi@canonical.com Signed-off-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org Signed-off-by: Jeremi Piotrowski jpiotrowski@linux.microsoft.com --- drivers/firmware/efi/libstub/arm64-stub.c | 39 ++++++++++++++++----- drivers/firmware/efi/libstub/efistub.h | 41 +++++++++++++++++++++-- drivers/firmware/efi/libstub/smbios.c | 13 +++++-- 3 files changed, 80 insertions(+), 13 deletions(-)
diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 42282c5c3fe6..e2f90566b291 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -17,20 +17,43 @@
static bool system_needs_vamap(void) { - const u8 *type1_family = efi_get_smbios_string(1, family); + const struct efi_smbios_type4_record *record; + const u32 __aligned(1) *socid; + const u8 *version;
/* * Ampere eMAG, Altra, and Altra Max machines crash in SetTime() if - * SetVirtualAddressMap() has not been called prior. + * SetVirtualAddressMap() has not been called prior. Most Altra systems + * can be identified by the SMCCC soc ID, which is conveniently exposed + * via the type 4 SMBIOS records. Otherwise, test the processor version + * field. eMAG systems all appear to have the processor version field + * set to "eMAG". */ - if (!type1_family || ( - strcmp(type1_family, "eMAG") && - strcmp(type1_family, "Altra") && - strcmp(type1_family, "Altra Max"))) + record = (struct efi_smbios_type4_record *)efi_get_smbios_record(4); + if (!record) return false;
- efi_warn("Working around broken SetVirtualAddressMap()\n"); - return true; + socid = (u32 *)record->processor_id; + switch (*socid & 0xffff000f) { + static char const altra[] = "Ampere(TM) Altra(TM) Processor"; + static char const emag[] = "eMAG"; + + default: + version = efi_get_smbios_string(&record->header, 4, + processor_version); + if (!version || (strncmp(version, altra, sizeof(altra) - 1) && + strncmp(version, emag, sizeof(emag) - 1))) + break; + + fallthrough; + + case 0x0a160001: // Altra + case 0x0a160002: // Altra Max + efi_warn("Working around broken SetVirtualAddressMap()\n"); + return true; + } + + return false; }
efi_status_t check_platform_features(void) diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 900df67a2078..970e86e3aab0 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -983,6 +983,8 @@ struct efi_smbios_record { u16 handle; };
+const struct efi_smbios_record *efi_get_smbios_record(u8 type); + struct efi_smbios_type1_record { struct efi_smbios_record header;
@@ -996,13 +998,46 @@ struct efi_smbios_type1_record { u8 family; };
-#define efi_get_smbios_string(__type, __name) ({ \ +struct efi_smbios_type4_record { + struct efi_smbios_record header; + + u8 socket; + u8 processor_type; + u8 processor_family; + u8 processor_manufacturer; + u8 processor_id[8]; + u8 processor_version; + u8 voltage; + u16 external_clock; + u16 max_speed; + u16 current_speed; + u8 status; + u8 processor_upgrade; + u16 l1_cache_handle; + u16 l2_cache_handle; + u16 l3_cache_handle; + u8 serial_number; + u8 asset_tag; + u8 part_number; + u8 core_count; + u8 enabled_core_count; + u8 thread_count; + u16 processor_characteristics; + u16 processor_family2; + u16 core_count2; + u16 enabled_core_count2; + u16 thread_count2; + u16 thread_enabled; +}; + +#define efi_get_smbios_string(__record, __type, __name) ({ \ int size = sizeof(struct efi_smbios_type ## __type ## _record); \ int off = offsetof(struct efi_smbios_type ## __type ## _record, \ __name); \ - __efi_get_smbios_string(__type, off, size); \ + __efi_get_smbios_string((__record), __type, off, size); \ })
-const u8 *__efi_get_smbios_string(u8 type, int offset, int recsize); +const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record, + u8 type, int offset, int recsize);
#endif diff --git a/drivers/firmware/efi/libstub/smbios.c b/drivers/firmware/efi/libstub/smbios.c index aadb422b9637..f9c159c28f46 100644 --- a/drivers/firmware/efi/libstub/smbios.c +++ b/drivers/firmware/efi/libstub/smbios.c @@ -22,19 +22,28 @@ struct efi_smbios_protocol { u8 minor_version; };
-const u8 *__efi_get_smbios_string(u8 type, int offset, int recsize) +const struct efi_smbios_record *efi_get_smbios_record(u8 type) { struct efi_smbios_record *record; efi_smbios_protocol_t *smbios; efi_status_t status; u16 handle = 0xfffe; - const u8 *strtable;
status = efi_bs_call(locate_protocol, &EFI_SMBIOS_PROTOCOL_GUID, NULL, (void **)&smbios) ?: efi_call_proto(smbios, get_next, &handle, &type, &record, NULL); if (status != EFI_SUCCESS) return NULL; + return record; +} + +const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record, + u8 type, int offset, int recsize) +{ + const u8 *strtable; + + if (!record) + return NULL;
strtable = (u8 *)record + record->length; for (int i = 1; i < ((u8 *)record)[offset]; i++) {
On Wed, Jun 07, 2023 at 05:26:12AM -0700, Ard Biesheuvel wrote:
[ Upstream commit eb684408f3ea4856639675d6465f0024e498e4b1 ]
Instead of using the SMBIOS type 1 record 'family' field, which is often modified by OEMs, use the type 4 'processor ID' and 'processor version' fields, which are set to a small set of probe-able values on all known Ampere EFI systems in the field.
Fixes: 550b33cfd4452968 ("arm64: efi: Force the use of ...") Tested-by: Andrea Righi andrea.righi@canonical.com Signed-off-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org Signed-off-by: Jeremi Piotrowski jpiotrowski@linux.microsoft.com
drivers/firmware/efi/libstub/arm64-stub.c | 39 ++++++++++++++++----- drivers/firmware/efi/libstub/efistub.h | 41 +++++++++++++++++++++-- drivers/firmware/efi/libstub/smbios.c | 13 +++++-- 3 files changed, 80 insertions(+), 13 deletions(-)
diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 42282c5c3fe6..e2f90566b291 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c
Sorry Ard, didn't meant to spoof your sender address.
Greg: commit 550b33cfd4452968 is in v6.1 but the fix for it (upstream eb684408f3ea4856639675d6465f0024e498e4b1) was not marked for stable. Hence this patch. It also needed a slight tweak because the file has been split since v6.1 (drivers/firmware/efi/libstub/arm64-stub.c => arm64.c).
My Ampere system returns "Server" to 'dmidecode -s system-family', so it hits this.
Thanks, Jeremi
On Wed, Jun 07, 2023 at 05:26:12AM -0700, Ard Biesheuvel wrote:
[ Upstream commit eb684408f3ea4856639675d6465f0024e498e4b1 ]
Instead of using the SMBIOS type 1 record 'family' field, which is often modified by OEMs, use the type 4 'processor ID' and 'processor version' fields, which are set to a small set of probe-able values on all known Ampere EFI systems in the field.
Fixes: 550b33cfd4452968 ("arm64: efi: Force the use of ...") Tested-by: Andrea Righi andrea.righi@canonical.com Signed-off-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Where did Sasha sign off on this?
Signed-off-by: Jeremi Piotrowski jpiotrowski@linux.microsoft.com
drivers/firmware/efi/libstub/arm64-stub.c | 39 ++++++++++++++++----- drivers/firmware/efi/libstub/efistub.h | 41 +++++++++++++++++++++-- drivers/firmware/efi/libstub/smbios.c | 13 +++++-- 3 files changed, 80 insertions(+), 13 deletions(-)
diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 42282c5c3fe6..e2f90566b291 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -17,20 +17,43 @@ static bool system_needs_vamap(void) {
- const u8 *type1_family = efi_get_smbios_string(1, family);
- const struct efi_smbios_type4_record *record;
- const u32 __aligned(1) *socid;
- const u8 *version;
/* * Ampere eMAG, Altra, and Altra Max machines crash in SetTime() if
* SetVirtualAddressMap() has not been called prior.
* SetVirtualAddressMap() has not been called prior. Most Altra systems* can be identified by the SMCCC soc ID, which is conveniently exposed* via the type 4 SMBIOS records. Otherwise, test the processor version* field. eMAG systems all appear to have the processor version field* set to "eMAG".
- if (!type1_family || (
strcmp(type1_family, "eMAG") &&strcmp(type1_family, "Altra") &&strcmp(type1_family, "Altra Max")))
- record = (struct efi_smbios_type4_record *)efi_get_smbios_record(4);
- if (!record) return false;
- efi_warn("Working around broken SetVirtualAddressMap()\n");
- return true;
- socid = (u32 *)record->processor_id;
- switch (*socid & 0xffff000f) {
static char const altra[] = "Ampere(TM) Altra(TM) Processor";static char const emag[] = "eMAG";- default:
version = efi_get_smbios_string(&record->header, 4,processor_version);if (!version || (strncmp(version, altra, sizeof(altra) - 1) &&strncmp(version, emag, sizeof(emag) - 1)))break;fallthrough;- case 0x0a160001: // Altra
- case 0x0a160002: // Altra Max
efi_warn("Working around broken SetVirtualAddressMap()\n");return true;- }
- return false;
} efi_status_t check_platform_features(void) diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 900df67a2078..970e86e3aab0 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -983,6 +983,8 @@ struct efi_smbios_record { u16 handle; }; +const struct efi_smbios_record *efi_get_smbios_record(u8 type);
struct efi_smbios_type1_record { struct efi_smbios_record header; @@ -996,13 +998,46 @@ struct efi_smbios_type1_record { u8 family; }; -#define efi_get_smbios_string(__type, __name) ({ \ +struct efi_smbios_type4_record {
- struct efi_smbios_record header;
- u8 socket;
- u8 processor_type;
- u8 processor_family;
- u8 processor_manufacturer;
- u8 processor_id[8];
- u8 processor_version;
- u8 voltage;
- u16 external_clock;
- u16 max_speed;
- u16 current_speed;
- u8 status;
- u8 processor_upgrade;
- u16 l1_cache_handle;
- u16 l2_cache_handle;
- u16 l3_cache_handle;
- u8 serial_number;
- u8 asset_tag;
- u8 part_number;
- u8 core_count;
- u8 enabled_core_count;
- u8 thread_count;
- u16 processor_characteristics;
- u16 processor_family2;
- u16 core_count2;
- u16 enabled_core_count2;
- u16 thread_count2;
- u16 thread_enabled;
+};
+#define efi_get_smbios_string(__record, __type, __name) ({ \ int size = sizeof(struct efi_smbios_type ## __type ## _record); \ int off = offsetof(struct efi_smbios_type ## __type ## _record, \ __name); \
- __efi_get_smbios_string(__type, off, size); \
- __efi_get_smbios_string((__record), __type, off, size); \
}) -const u8 *__efi_get_smbios_string(u8 type, int offset, int recsize); +const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record,
u8 type, int offset, int recsize);#endif diff --git a/drivers/firmware/efi/libstub/smbios.c b/drivers/firmware/efi/libstub/smbios.c index aadb422b9637..f9c159c28f46 100644 --- a/drivers/firmware/efi/libstub/smbios.c +++ b/drivers/firmware/efi/libstub/smbios.c @@ -22,19 +22,28 @@ struct efi_smbios_protocol { u8 minor_version; }; -const u8 *__efi_get_smbios_string(u8 type, int offset, int recsize) +const struct efi_smbios_record *efi_get_smbios_record(u8 type) { struct efi_smbios_record *record; efi_smbios_protocol_t *smbios; efi_status_t status; u16 handle = 0xfffe;
- const u8 *strtable;
status = efi_bs_call(locate_protocol, &EFI_SMBIOS_PROTOCOL_GUID, NULL, (void **)&smbios) ?: efi_call_proto(smbios, get_next, &handle, &type, &record, NULL); if (status != EFI_SUCCESS) return NULL;
- return record;
+}
+const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record,
u8 type, int offset, int recsize)+{
- const u8 *strtable;
- if (!record)
return NULL;strtable = (u8 *)record + record->length; for (int i = 1; i < ((u8 *)record)[offset]; i++) { -- 2.30.2
On 6/7/2023 8:36 PM, Greg KH wrote:
On Wed, Jun 07, 2023 at 05:26:12AM -0700, Ard Biesheuvel wrote:
[ Upstream commit eb684408f3ea4856639675d6465f0024e498e4b1 ]
Instead of using the SMBIOS type 1 record 'family' field, which is often modified by OEMs, use the type 4 'processor ID' and 'processor version' fields, which are set to a small set of probe-able values on all known Ampere EFI systems in the field.
Fixes: 550b33cfd4452968 ("arm64: efi: Force the use of ...") Tested-by: Andrea Righi andrea.righi@canonical.com Signed-off-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Where did Sasha sign off on this?
I must have picked the commit from the 6.2 backport:
https://lore.kernel.org/stable/20230328142621.544265000@linuxfoundation.org/... https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v...
On Thu, Jun 08, 2023 at 09:36:22AM +0200, Jeremi Piotrowski wrote:
On 6/7/2023 8:36 PM, Greg KH wrote:
On Wed, Jun 07, 2023 at 05:26:12AM -0700, Ard Biesheuvel wrote:
[ Upstream commit eb684408f3ea4856639675d6465f0024e498e4b1 ]
Instead of using the SMBIOS type 1 record 'family' field, which is often modified by OEMs, use the type 4 'processor ID' and 'processor version' fields, which are set to a small set of probe-able values on all known Ampere EFI systems in the field.
Fixes: 550b33cfd4452968 ("arm64: efi: Force the use of ...") Tested-by: Andrea Righi andrea.righi@canonical.com Signed-off-by: Ard Biesheuvel ardb@kernel.org Signed-off-by: Sasha Levin sashal@kernel.org
Where did Sasha sign off on this?
I must have picked the commit from the 6.2 backport:
https://lore.kernel.org/stable/20230328142621.544265000@linuxfoundation.org/... https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v...
When doing so, please be explicit, otherwise it is very confusing.
thanks,
greg k-h
linux-stable-mirror@lists.linaro.org
-
Ard Biesheuvel -
Greg KH -
Jeremi Piotrowski