Hi,
After updating to 6.14.2, the ethernet adapter is almost unusable, I get over 30% packet loss. Bisect says it's this commit:
commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b Author: Vitaly Lifshits vitaly.lifshits@intel.com Date: Thu Mar 13 16:05:56 2025 +0200
e1000e: change k1 configuration on MTP and later platforms
[ Upstream commit efaaf344bc2917cbfa5997633bc18a05d3aed27f ]
Starting from Meteor Lake, the Kumeran interface between the integrated MAC and the I219 PHY works at a different frequency. This causes sporadic MDI errors when accessing the PHY, and in rare circumstances could lead to packet corruption.
To overcome this, introduce minor changes to the Kumeran idle state (K1) parameters during device initialization. Hardware reset reverts this configuration, therefore it needs to be applied in a few places.
Fixes: cc23f4f0b6b9 ("e1000e: Add support for Meteor Lake") Signed-off-by: Vitaly Lifshits vitaly.lifshits@intel.com Tested-by: Avigail Dahan avigailx.dahan@intel.com Signed-off-by: Tony Nguyen anthony.l.nguyen@intel.com Signed-off-by: Sasha Levin sashal@kernel.org
drivers/net/ethernet/intel/e1000e/defines.h | 3 ++ drivers/net/ethernet/intel/e1000e/ich8lan.c | 80 +++++++++++++++++++++++++++-- drivers/net/ethernet/intel/e1000e/ich8lan.h | 4 ++ 3 files changed, 82 insertions(+), 5 deletions(-)
My system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And the e1000e driver is running in a Xen HVM (with PCI passthrough). Interestingly, I have also another one with Intel Core Ultra 7 155H where the issue does not happen. I don't see what is different about network adapter there, they look identical on lspci (but there are differences about other devices)...
I see the commit above was already backported to other stable branches too...
#regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b
On 4/14/2025 3:18 PM, Marek Marczykowski-Górecki wrote:
Hi,
After updating to 6.14.2, the ethernet adapter is almost unusable, I get over 30% packet loss. Bisect says it's this commit:
commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b Author: Vitaly Lifshits <vitaly.lifshits@intel.com> Date: Thu Mar 13 16:05:56 2025 +0200 e1000e: change k1 configuration on MTP and later platformsMy system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And the e1000e driver is running in a Xen HVM (with PCI passthrough). Interestingly, I have also another one with Intel Core Ultra 7 155H where the issue does not happen. I don't see what is different about network adapter there, they look identical on lspci (but there are differences about other devices)...
I see the commit above was already backported to other stable branches too...
#regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b
Thank you for this report.
Do you see the high packet loss without the virtualization? Can you please share the lspci output? Does your switch/link partner support flow control? if it is configurable can you try to enable it? Do you see any errors in dmesg related to the e1000e driver?
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Can you please share the lspci output?
Sure:
00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20) Subsystem: CLEVO/KAPOK Computer Device [1558:a743] Physical Slot: 7 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin D routed to IRQ 69 Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K] Capabilities: [c8] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Kernel driver in use: e1000e Kernel modules: e1000e
Does your switch/link partner support flow control? if it is configurable can you try to enable it?
It does support it. Enabling it makes things much worse...
Do you see any errors in dmesg related to the e1000e driver?
Not really. dmesg | grep 'e1000e|ens7':
[ 3.088489] e1000e: Intel(R) PRO/1000 Network Driver [ 3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock [ 3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb [ 3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection [ 3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF [ 3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0 [ 8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down [ 242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
(you can also see a test with flow control above)
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
On 4/14/2025 3:58 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Can you please share the lspci output?
Sure:
00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20) Subsystem: CLEVO/KAPOK Computer Device [1558:a743] Physical Slot: 7 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin D routed to IRQ 69 Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K] Capabilities: [c8] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Kernel driver in use: e1000e Kernel modules: e1000e
Do you have mei modules running? Can you try if disabling them make things better?
Does your switch/link partner support flow control? if it is configurable can you try to enable it?
It does support it. Enabling it makes things much worse...
Do you see any errors in dmesg related to the e1000e driver?
Not really. dmesg | grep 'e1000e|ens7':
[ 3.088489] e1000e: Intel(R) PRO/1000 Network Driver [ 3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock [ 3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb [ 3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection [ 3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF [ 3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0 [ 8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down [ 242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
(you can also see a test with flow control above)
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Dear Marek,
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
Please inform us if disabling the CPU low power states resolves the issue or if further investigation is required. As previously mentioned, this patch is critical for the operation of Meteor Lake LAN devices, and therefore, we are unable to revert it.
Thank you for your cooperation.
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly wrote:
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Dear Marek,
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > Can you please also share the output of ethtool -i? I would like to know the > NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
Kind regards,
Paul
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > > Can you please also share the output of ethtool -i? I would like to know the > > NVM version that you have on your device. > > driver: e1000e > version: 6.14.1+ > firmware-version: 1.1-4 > expansion-rom-version: > bus-info: 0000:00:1f.6 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes >
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote: > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > > > Can you please also share the output of ethtool -i? I would like to know the > > > NVM version that you have on your device. > > > > driver: e1000e > > version: 6.14.1+ > > firmware-version: 1.1-4 > > expansion-rom-version: > > bus-info: 0000:00:1f.6 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > > > Your firmware version is not the latest, can you check with the board > manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
> Also, you mentioned that on another system this issue doesn't reproduce, do > they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
Same with 6.15-rc5.
On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote: > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote: > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > > > > Can you please also share the output of ethtool -i? I would like to know the > > > > NVM version that you have on your device. > > > > > > driver: e1000e > > > version: 6.14.1+ > > > firmware-version: 1.1-4 > > > expansion-rom-version: > > > bus-info: 0000:00:1f.6 > > > supports-statistics: yes > > > supports-test: yes > > > supports-eeprom-access: yes > > > supports-register-dump: yes > > > supports-priv-flags: yes > > > > > > > Your firmware version is not the latest, can you check with the board > > manufacturer if there is a BIOS update to your system? > > I can check, but still, it's a regression in the Linux driver - old > kernel did work perfectly well on this hw. Maybe new driver tries to use > some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
> > Also, you mentioned that on another system this issue doesn't reproduce, do > > they have the same firmware version? > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from > 6.14.2 works fine there.
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
Same with 6.15-rc5.
And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has this buggy patch revered and nobody complained (contrary to the version with the patch included). Should I submit the revert patch?
On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote: > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote: > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote: > > > > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > > > > > Can you please also share the output of ethtool -i? I would like to know the > > > > > NVM version that you have on your device. > > > > > > > > driver: e1000e > > > > version: 6.14.1+ > > > > firmware-version: 1.1-4 > > > > expansion-rom-version: > > > > bus-info: 0000:00:1f.6 > > > > supports-statistics: yes > > > > supports-test: yes > > > > supports-eeprom-access: yes > > > > supports-register-dump: yes > > > > supports-priv-flags: yes > > > > > > > > > > Your firmware version is not the latest, can you check with the board > > > manufacturer if there is a BIOS update to your system? > > > > I can check, but still, it's a regression in the Linux driver - old > > kernel did work perfectly well on this hw. Maybe new driver tries to use > > some feature that is missing (or broken) in the old firmware? > > A little bit of context: I'm maintaining the kernel package for a Qubes > OS distribution. While I can try to update firmware on my test system, I > have no influence on what hardware users will use this kernel, and > which firmware version they will use (and whether all the vendors > provide newer firmware at all). I cannot ship a kernel that is known > to break network on some devices. > > > > Also, you mentioned that on another system this issue doesn't reproduce, do > > > they have the same firmware version? > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from > > 6.14.2 works fine there.
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
Same with 6.15-rc5.
And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has this buggy patch revered and nobody complained (contrary to the version with the patch included). Should I submit the revert patch?
Just submit a revert then 👍 I have no authority here, but had good experience with just sending a revert patch in the past 🤗
Cheers, Chris
On 6/18/2025 4:41 PM, Christian Heusel wrote:
On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote: >> On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote: >>> On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote: >>>> >>>> >>>> On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: >>>>> On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: >>>>>> Can you please also share the output of ethtool -i? I would like to know the >>>>>> NVM version that you have on your device. >>>>> >>>>> driver: e1000e >>>>> version: 6.14.1+ >>>>> firmware-version: 1.1-4 >>>>> expansion-rom-version: >>>>> bus-info: 0000:00:1f.6 >>>>> supports-statistics: yes >>>>> supports-test: yes >>>>> supports-eeprom-access: yes >>>>> supports-register-dump: yes >>>>> supports-priv-flags: yes >>>>> >>>> >>>> Your firmware version is not the latest, can you check with the board >>>> manufacturer if there is a BIOS update to your system? >>> >>> I can check, but still, it's a regression in the Linux driver - old >>> kernel did work perfectly well on this hw. Maybe new driver tries to use >>> some feature that is missing (or broken) in the old firmware? >> >> A little bit of context: I'm maintaining the kernel package for a Qubes >> OS distribution. While I can try to update firmware on my test system, I >> have no influence on what hardware users will use this kernel, and >> which firmware version they will use (and whether all the vendors >> provide newer firmware at all). I cannot ship a kernel that is known >> to break network on some devices. >> >>>> Also, you mentioned that on another system this issue doesn't reproduce, do >>>> they have the same firmware version? >>> >>> The other one has also 1.1-4 firmware. And I re-checked, e1000e from >>> 6.14.2 works fine there.
> Thank you for your detailed feedback and for providing the requested > information. > > We have conducted extensive testing of this patch across multiple systems > and have not observed any packet loss issues. Upon comparing the mentioned > setups, we noted that while the LAN controller is similar, the CPU differs. > We believe that the issue may be related to transitions in the CPU's low > power states. > > Consequently, we kindly request that you disable the CPU low power state > transitions in the S0 system state and verify if the issue persists. You can > disable this in the kernel parameters on the command line with idle=poll. > Please note that this command is intended for debugging purposes only, as it > may result in higher power consumption.
I tried with idle=poll, and it didn't help, I still see a lot of packet losses. But I can also confirm that idle=poll makes the system use significantly more power (previously at 25-30W, with this option stays at about 42W).
Is there any other info I can provide, enable some debug features or something?
I see the problem is with receiving packets - in my simple ping test, the ping target sees all the echo requests (and respond to them), but the responses aren't reaching ping back (and are not visible on tcpdump on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
Same with 6.15-rc5.
And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has this buggy patch revered and nobody complained (contrary to the version with the patch included). Should I submit the revert patch?
It is not a good idea to revert this patch as most of the systems will encounter the original issues (PHY access and packet loss). The reason I first introduced this patch was because big vendors reported the packet loss issue. You can refer to the following sightings: https://answers.launchpad.net/ubuntu/+question/816003 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2066064 https://bugzilla.kernel.org/show_bug.cgi?id=218869
As an intermediate solution we can either use a privileged flag to make it configurable. I will share with you a patch that might fix the issue on your system that I would like you to try.
FYI, we are currently investigating a similar issue that seems to be due to a misconfiguration of the system firmware.
Just submit a revert then 👍 I have no authority here, but had good experience with just sending a revert patch in the past 🤗
Cheers, Chris
On Thu, Jun 19, 2025 at 03:20:35PM +0300, Lifshits, Vitaly wrote:
On 6/18/2025 4:41 PM, Christian Heusel wrote:
On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
Dear Marek, dear Vitaly,
Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki: > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote: > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote: > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote: > > > > > > > > > > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote: > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote: > > > > > > > Can you please also share the output of ethtool -i? I would like to know the > > > > > > > NVM version that you have on your device. > > > > > > > > > > > > driver: e1000e > > > > > > version: 6.14.1+ > > > > > > firmware-version: 1.1-4 > > > > > > expansion-rom-version: > > > > > > bus-info: 0000:00:1f.6 > > > > > > supports-statistics: yes > > > > > > supports-test: yes > > > > > > supports-eeprom-access: yes > > > > > > supports-register-dump: yes > > > > > > supports-priv-flags: yes > > > > > > > > > > > > > > > > Your firmware version is not the latest, can you check with the board > > > > > manufacturer if there is a BIOS update to your system? > > > > > > > > I can check, but still, it's a regression in the Linux driver - old > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use > > > > some feature that is missing (or broken) in the old firmware? > > > > > > A little bit of context: I'm maintaining the kernel package for a Qubes > > > OS distribution. While I can try to update firmware on my test system, I > > > have no influence on what hardware users will use this kernel, and > > > which firmware version they will use (and whether all the vendors > > > provide newer firmware at all). I cannot ship a kernel that is known > > > to break network on some devices. > > > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do > > > > > they have the same firmware version? > > > > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from > > > > 6.14.2 works fine there.
> > Thank you for your detailed feedback and for providing the requested > > information. > > > > We have conducted extensive testing of this patch across multiple systems > > and have not observed any packet loss issues. Upon comparing the mentioned > > setups, we noted that while the LAN controller is similar, the CPU differs. > > We believe that the issue may be related to transitions in the CPU's low > > power states. > > > > Consequently, we kindly request that you disable the CPU low power state > > transitions in the S0 system state and verify if the issue persists. You can > > disable this in the kernel parameters on the command line with idle=poll. > > Please note that this command is intended for debugging purposes only, as it > > may result in higher power consumption. > > I tried with idle=poll, and it didn't help, I still see a lot of packet > losses. But I can also confirm that idle=poll makes the system use > significantly more power (previously at 25-30W, with this option stays > at about 42W). > > Is there any other info I can provide, enable some debug features or > something? > > I see the problem is with receiving packets - in my simple ping test, > the ping target sees all the echo requests (and respond to them), but > the responses aren't reaching ping back (and are not visible on tcpdump > on the problematic system either).
As the cause is still unclear, can the commit please be reverted in the master branch due adhere to Linux’ no-regression policy, so that it can be reverted from the stable series?
Marek, did you also test 6.15 release candidates?
The last test I did was on 6.15-rc3. I can re-test on -rc5.
Same with 6.15-rc5.
And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has this buggy patch revered and nobody complained (contrary to the version with the patch included). Should I submit the revert patch?
It is not a good idea to revert this patch as most of the systems will encounter the original issues (PHY access and packet loss). The reason I first introduced this patch was because big vendors reported the packet loss issue. You can refer to the following sightings: https://answers.launchpad.net/ubuntu/+question/816003 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2066064 https://bugzilla.kernel.org/show_bug.cgi?id=218869
It would be useful to have any of those links in the original commit...
As an intermediate solution we can either use a privileged flag to make it configurable. I will share with you a patch that might fix the issue on your system that I would like you to try.
Yes, that patch works :)
FYI, we are currently investigating a similar issue that seems to be due to a misconfiguration of the system firmware.
Can you share some details? I can forward the info to firmware developers for this system (it's Dasharo - coreboot-based firmware).
On Mon, Apr 14, 2025 at 02:58:09PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Tried now, the same issue is without any virtualization too.
linux-stable-mirror@lists.linaro.org
-
Christian Heusel -
Lifshits, Vitaly -
Marek Marczykowski-Górecki -
Paul Menzel