Hi Mario,

I apologize. I think I mixed up the versions between linux-lts and linux kernel.

linux-6.6.28-1-lts works: https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5 linux-6.6.57-1-lts works: https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7

linux-6.11.4-arch2-1 doesn't work: https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work: https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388

Kind regards,

Rick

On 22-10-2024 14:55, Mario Limonciello wrote:

On 10/22/2024 07:13, rick@581238.xyz wrote:

...

Hi all,

I am having the exact same issue.

linux-lts-6.6.28-1 works, anything above doesn't.

When kernel above linux-lts-6.6.28-1:

• Boltctl does not show anything
• thunderbolt.host_reset=0 had no impact
• triggers following errors:

[   50.627948] ucsi_acpi USBC000:00: unknown error 0    [   50.627957] ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-5)

Gists:

• https://gist.github.com/ricklahaye/83695df8c8273c30d2403da97a353e15

dmesg with "Linux system 6.11.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 17 Oct 2024 20:53:41 +0000 x86_64 GNU/Linux" where thunderbolt dock does not work

• https://gist.github.com/ricklahaye/79e4040abcd368524633e86addec1833

dmesg with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work

• https://gist.github.com/ricklahaye/c9a7b4a7eeba5e7900194eecf9fce454

boltctl with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work

Kind regards, Rick

Ps: sorry for resend; this time with plain text format

Can you please share a log with 'thunderbolt.host_reset=0 thunderbolt.dyndbg' on the kernel command line in a kernel that it doesn't work?  This should make the behavior match 6.6.28 and we can compare.

Maybe the best thing would be:

• 6.6.28 w/ thunderbolt.dyndbg
• 6.6.29 w/ thunderbolt.dyndbg thunderbolt.host_reset=0

On 10/22/2024 11:10, Mika Westerberg wrote:

Hi,

On Tue, Oct 22, 2024 at 05:44:18PM +0200, Rick wrote:

...

Hi Mario,

I apologize. I think I mixed up the versions between linux-lts and linux kernel.

linux-6.6.28-1-lts works: https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5 linux-6.6.57-1-lts works: https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7

linux-6.11.4-arch2-1 doesn't work: https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work: https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388

Looks like some sort of connectivity issue to me.

However, can you first drop the "pcie_aspm=force" from the command line and see if that has any affect. Probably does not but I suggest not to keep it unless you really know that you want force ASPM on all PCIe links (this may cause issues with some devices).

Anyways, even the working -lts ones you see the link goes down and up which is not expected (unless you unplugged and plugged it back). Some devices that are not Thunderbolt certified have this kinds of issues. The cable could make difference too. Is it Thunderbolt 4 cable or regular one?

And if it's really starting to fail consistently 6.6.29, the number of commits is very small, it should be a pretty quick bisect.

❯ git log --oneline v6.6.28..v6.6.29 | wc -l 158

Hi,

On Tue, Oct 22, 2024 at 07:06:50PM +0200, Rick wrote:

Hi Mika,

I have removed pcie_asm=force as kernel parameter but still not working on latest non LTS kernel.

Okay, I still suggest not having that unless you absolutely know that you need it.

In regards to the disconnect; sorry I think I might have turned of the docking station myself during that test. I have taken another dmesg without me disconnecting the docking station: https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1

The cable is the original Thunderbolt 4 cable that came with the docking station. I have used it on this laptop using Windows (dualboot) without any issues. Also on another Windows laptop also without issues. It was used in 40Gbit mode.

In the dmesg you shared above, there are still unplug and USB tunnel creation fails so you only get USB 2.x connection with all the USB devices on the dock.

How do you determine if it "works"? I guess keyboard and mouse (both USB 2.x devices) and display (tunneled over USB4 link) all are working right? However, if you plug in USB 3.x device to the dock it enumerates as FullSpeed instead of SuperSpeed. There is definitely something wrong here. I asked from our TB validation folks if they have any experience with this dock but did not receive any reply yet.

What you mean by 40Gbit mode? The dock exposes two lanes both at 20G so it should always be 40G since we bind the lanes, also in Windows.

Also In Windows, do you see if the all USB devices on the dock are enumerated as FullSpeed or SuperSpeed? I suspect it's the former there too but can you check? Keyboard and mouse should be FullSpeed but there is some audio device that may be USB 3.x (SuperSpeed), or alternatively if you have USB 3.x memory stick (or any other device) you can plug that to the dock and see how it enumerates.

Hi,

On Fri, Oct 25, 2024 at 12:20:55PM +0200, Rick wrote:

Hi Mika

On 23-10-2024 08:10, Mika Westerberg wrote:

...

Hi,

On Tue, Oct 22, 2024 at 07:06:50PM +0200, Rick wrote:

...

Hi Mika,

I have removed pcie_asm=force as kernel parameter but still not working on latest non LTS kernel.

Okay, I still suggest not having that unless you absolutely know that you need it.

Noted thank you!

...

...

In regards to the disconnect; sorry I think I might have turned of the docking station myself during that test. I have taken another dmesg without me disconnecting the docking station: https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1

The cable is the original Thunderbolt 4 cable that came with the docking station. I have used it on this laptop using Windows (dualboot) without any issues. Also on another Windows laptop also without issues. It was used in 40Gbit mode.

In the dmesg you shared above, there are still unplug and USB tunnel creation fails so you only get USB 2.x connection with all the USB devices on the dock.

Yes you are right. I removed all attached USB devices from the dock, but still see "3:3: USB3 tunnel activation failed, aborting"

...

How do you determine if it "works"? I guess keyboard and mouse (both USB 2.x devices) and display (tunneled over USB4 link) all are working right? However, if you plug in USB 3.x device to the dock it enumerates as FullSpeed instead of SuperSpeed. There is definitely something wrong here. I asked from our TB validation folks if they have any experience with this dock but did not receive any reply yet.

What you mean by 40Gbit mode? The dock exposes two lanes both at 20G so it should always be 40G since we bind the lanes, also in Windows.

2 lanes of 20G indeed.

...

Also In Windows, do you see if the all USB devices on the dock are enumerated as FullSpeed or SuperSpeed? I suspect it's the former there too but can you check? Keyboard and mouse should be FullSpeed but there is some audio device that may be USB 3.x (SuperSpeed), or alternatively if you have USB 3.x memory stick (or any other device) you can plug that to the dock and see how it enumerates.

I checked on Windows with some 3.1 USB devices, and they were properly seen as 3.1 Superspeed+/10Gbps when attached to dock (using USBView from Windows SDK).

I also tried some Linux kernels, and it seems that 6.9 works, and 6.10 doesn't.

6.9: https://gist.github.com/ricklahaye/da8c63edb0c27dc55bef351f9f4dd035

I still see similar issue even with the v6.9 kernel. The link goes up an down unexpectly.

I wonder if you could try to take traces of the control channel traffic? I suggest to use v6.11 kernel because it should have all the tracing bits added then install tbtools [1] and follow the steps here:

https://github.com/intel/tbtools?tab=readme-ov-file#tracing

Then provide both full dmesg and the trace output. That hopefully shows if some of the access we are doing in the Linux side is causing the link to to drop. Let me know if you need more detailed instructions.

Also please drop the "thunderbolt.host_reset=0" from the command line as that did not help, so it is not needed.

[1] https://github.com/intel/tbtools

Hi Rick,

On Wed, Oct 30, 2024 at 08:11:30AM +0100, Rick wrote:

Hi Mika,

Thank you for your email.

On 28-10-2024 09:18, Mika Westerberg wrote:

...

I still see similar issue even with the v6.9 kernel. The link goes up an down unexpectly.

I wonder if you could try to take traces of the control channel traffic? I suggest to use v6.11 kernel because it should have all the tracing bits added then install tbtools [1] and follow the steps here:

https://github.com/intel/tbtools?tab=readme-ov-file#tracing

Then provide both full dmesg and the trace output. That hopefully shows if some of the access we are doing in the Linux side is causing the link to to drop. Let me know if you need more detailed instructions.

Also please drop the "thunderbolt.host_reset=0" from the command line as that did not help, so it is not needed.

Dropped thank you.

...

[1] https://github.com/intel/tbtools

tbtrace on 6.11.5: https://gist.github.com/ricklahaye/69776e9c39fd30a80e2adb6156bdb42d dmesg on 6.11.5: https://gist.github.com/ricklahaye/8588450725695a0bd45799d3d66c7aff

Thanks! I suspect there is something we do when we read the sideband that makes the device router to "timeout" and retry the link establishment. There is also the failure when USB 3.x tunnel is created but we can look that after we figure out the connection issue.

Looking at the trace we are still polling for retimers when we see the unplug:

[ 48.684078] tb_tx Read Request Domain 0 Route 0 Adapter 3 / Lane 0x00/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String High 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low 0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001 .... [00:12] 0x91 Address [13:18] 0x1 Read Size [19:24] 0x3 Adapter Num [25:26] 0x1 Configuration Space (CS) → Adapter Configuration Space [27:28] 0x0 Sequence Number (SN) [ 48.684339] tb_rx Read Response Domain 0 Route 0 Adapter 3 / Lane 0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000 .... Route String High 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low 0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001 .... [00:12] 0x91 Address [13:18] 0x1 Read Size [19:24] 0x3 Adapter Num [25:26] 0x1 Configuration Space (CS) → Adapter Configuration Space [27:28] 0x0 Sequence Number (SN) 0x03/0091 0x81320408 0b10000001 00110010 00000100 00001000 .2.. PORT_CS_1 [00:07] 0x8 Address [08:15] 0x4 Length [16:18] 0x2 Target [20:23] 0x3 Re-timer Index [24:24] 0x1 WnR [25:25] 0x0 No Response (NR) [26:26] 0x0 Result Code (RC) [31:31] 0x1 Pending (PND) [ 48.691410] tb_event Hot Plug Event Packet Domain 0 Route 0 Adapter 3 / Lane 0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000 .... Route String High 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low 0x02/---- 0x80000003 0b10000000 00000000 00000000 00000011 .... [00:05] 0x3 Adapter Num [31:31] 0x1 UPG [ 48.691414] thunderbolt 0000:00:0d.2: acking hot unplug event on 0:3

Taking this into account and also the fact that your previous email you say that v6.9 works and v6.10 does not, I wonder if you could first try just to revert:

c6ca1ac9f472 ("thunderbolt: Increase sideband access polling delay")

and see if that helps with the connection issue? If it does then can you take full dmesg and the trace again and with me so I can look at the USB tunnel issue too?