On Fri, Nov 07, 2025 at 05:53:51PM +0800, chuang wrote:
Thanks for reviewing the patch. I'm providing the detailed analysis and debugging traces below to confirm the root cause and exact location of the reference count leak.
- Environment and Symptom
The issue was consistently reproduced when routing TCP traffic through a Software IP Tunnel interface (sit0). The traffic flow is:
APP -> sit0 (IP tunnel) -> outside
This leads to a reference count leak that prevents the device from being freed during unregistration, resulting in the kernel log warning:
unregister_netdevice: waiting for sit0 to become free. Usage count = N
- Enable refcnt_tracer
Live-crash analysis identified a stale dst entry retaining a reference to sit0. With CONFIG_NET_DEV_REFCNT_TRACKER enabled, the allocation stack for the leaked reference was identified:
[1279559.416854] leaked reference. [1279559.416955] dst_init+0x48/0x100 [1279559.416965] dst_alloc+0x66/0xd0 [1279559.416966] rt_dst_alloc+0x3c/0xd0 [1279559.416974] ip_route_output_key_hash_rcu+0x1d7/0x940 [1279559.416978] ip_route_output_key_hash+0x6d/0xa0 [1279559.416979] ip_route_output_flow+0x1f/0x70 [1279559.416980] __ip_queue_xmit+0x415/0x480 [1279559.416984] ip_queue_xmit+0x15/0x20 [1279559.416986] __tcp_transmit_skb+0xad4/0xc50
- Pinpointing the Unmatched dst_hold()
To pinpoint the specific reference not released, we added tracepoints to all dst_hold/put functions and used eBPF to record the full lifecycle. The tracing identified a hold operation with the following call stack:
do_trace_dst_entry_inc+0x45 rt_set_nexthop.constprop.0+0x376 /* <<<<<<<<<<<<<<<<<<<< HERE */ __mkroute_output+0x2B7 ip_route_output_key_hash_rcu+0xBD ip_route_output_key_hash+0x6D ip_route_output_flow+0x1F inet_sk_rebuild_header+0x19C __tcp_retransmit_skb+0x7E tcp_retransmit_skb+0x19 tcp_retransmit_timer+0x3DF
The address rt_set_nexthop.constprop.0+0x376 corresponds to the dst_hold() call inside rt_bind_exception().
- Root Cause Analysis
The sit driver's packet transmission path calls: sit_tunnel_xmit() -> ... -> update_or_create_fnhe(), which lead to fnhe_remove_oldest() being called to delete entries exceeding the FNHE_RECLAIM_DEPTH+random.
The race window is between fnhe_remove_oldest() selecting fnheX for deletion and the subsequent kfree_rcu(). During this time, the concurrent path's __mkroute_output() -> find_exception() can fetch the soon-to-be-deleted fnheX, and rt_bind_exception() then binds it with a new dst using a dst_hold(). When the original fnheX is freed via RCU, the dst reference remains permanently leaked.
- Fix Validation with eBPF
The patch mitigates this by zeroing fnhe_daddr before the RCU-protected deletion steps. This prevents rt_bind_exception() from attempting to reuse the entry. The fix was validated by probing the rt_bind_exception path (which in my environment is optimized to rt_set_nexthop.constprop.0) to catch any zeroed but active FNHEs being processed:
bpftrace -e 'kprobe:rt_set_nexthop.constprop.0 { $rt = (struct rtable *)arg0; $fnhe = (struct fib_nh_exception *)arg3; $fi = (struct flowi *)arg4;
/* Check for an FNHE that is marked for deletion (daddr == 0) * but is still visible/valid (fnhe_expires != 0 and not expired). */ if ($fi != 0 && $fnhe != 0 && $fnhe->fnhe_daddr == 0 &&$fnhe->fnhe_expires != 0 && $fnhe->fnhe_expires >= jiffies) { printf("rt: %llx, dev: %s, will leak before this patch\n", $rt, $rt->dst.dev->name); } }'
Thanks for the details. I was able to reproduce the issue with [1] and I can confirm that it does not reproduce with your fix.
Are you going to submit v2?
[1] #!/bin/bash
ip netns add ns1 ip -n ns1 link set dev lo up ip -n ns1 address add 192.0.2.1/32 dev lo ip -n ns1 link add name dummy1 up type dummy ip -n ns1 route add 192.0.2.2/32 dev dummy1 ip -n ns1 link add name gretap1 up arp off type gretap local 192.0.2.1 remote 192.0.2.2 ip -n ns1 route add 198.51.0.0/16 dev gretap1 taskset -c 0 ip netns exec ns1 mausezahn gretap1 -A 198.51.100.1 -B 198.51.0.0/16 -t udp -p 1000 -c 0 -q & taskset -c 2 ip netns exec ns1 mausezahn gretap1 -A 198.51.100.1 -B 198.51.0.0/16 -t udp -p 1000 -c 0 -q & sleep 10 ip netns pids ns1 | xargs kill ip netns del ns1