This addresses the infamous unregister_netdevice splat in net selftests; the actual fix is carried by the first patch, while the 2nd one addresses a related problem in the relevant test that was patially hiding the problem.
Targeting net-next as the issue is quite old and I feel a little lost in the fib info/nh jungle. --- v1 -> v2: - drop unintended whitespace change in patch 1/2
Paolo Abeni (2): ipv6: release nexthop on device removal selftests: net: really check for bg process completion
net/ipv6/route.c | 6 +++--- tools/testing/selftests/net/pmtu.sh | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at dst_init+0x84/0x4a0 dst_alloc+0x97/0x150 ip6_dst_alloc+0x23/0x90 ip6_rt_pcpu_alloc+0x1e6/0x520 ip6_pol_route+0x56f/0x840 fib6_rule_lookup+0x334/0x630 ip6_route_output_flags+0x259/0x480 ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 ip6_dst_lookup_flow+0x88/0x190 udp_tunnel6_dst_lookup+0x2a7/0x4c0 vxlan_xmit_one+0xbde/0x4a50 [vxlan] vxlan_xmit+0x9ad/0xf20 [vxlan] dev_hard_start_xmit+0x10e/0x360 __dev_queue_xmit+0xf95/0x18c0 arp_solicit+0x4a2/0xe00 neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:
./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") Signed-off-by: Paolo Abeni pabeni@redhat.com --- v1 -> v2: - dropped unintended whitespace change --- net/ipv6/route.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/net/ipv6/route.c b/net/ipv6/route.c index d7ce5cf2017a..038c1eeef0be 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -374,6 +374,7 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev) { struct rt6_info *rt = dst_rt6_info(dst); struct inet6_dev *idev = rt->rt6i_idev; + struct fib6_info *from;
if (idev && idev->dev != blackhole_netdev) { struct inet6_dev *blackhole_idev = in6_dev_get(blackhole_netdev); @@ -383,6 +384,8 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev) in6_dev_put(idev); } } + from = unrcu_pointer(xchg(&rt->from, NULL)); + fib6_info_release(from); }
static bool __rt6_check_expired(const struct rt6_info *rt) @@ -1455,7 +1458,6 @@ static DEFINE_SPINLOCK(rt6_exception_lock); static void rt6_remove_exception(struct rt6_exception_bucket *bucket, struct rt6_exception *rt6_ex) { - struct fib6_info *from; struct net *net;
if (!bucket || !rt6_ex) @@ -1467,8 +1469,6 @@ static void rt6_remove_exception(struct rt6_exception_bucket *bucket, /* purge completely the exception to allow releasing the held resources: * some [sk] cache may keep the dst around for unlimited time */ - from = unrcu_pointer(xchg(&rt6_ex->rt6i->from, NULL)); - fib6_info_release(from); dst_dev_put(&rt6_ex->rt6i->dst);
hlist_del_rcu(&rt6_ex->hlist);
On Tue, Nov 5, 2024 at 7:24 PM Paolo Abeni pabeni@redhat.com wrote:
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at dst_init+0x84/0x4a0 dst_alloc+0x97/0x150 ip6_dst_alloc+0x23/0x90 ip6_rt_pcpu_alloc+0x1e6/0x520 ip6_pol_route+0x56f/0x840 fib6_rule_lookup+0x334/0x630 ip6_route_output_flags+0x259/0x480 ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 ip6_dst_lookup_flow+0x88/0x190 udp_tunnel6_dst_lookup+0x2a7/0x4c0 vxlan_xmit_one+0xbde/0x4a50 [vxlan] vxlan_xmit+0x9ad/0xf20 [vxlan] dev_hard_start_xmit+0x10e/0x360 __dev_queue_xmit+0xf95/0x18c0 arp_solicit+0x4a2/0xe00 neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:
./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") Signed-off-by: Paolo Abeni pabeni@redhat.com
Thanks a lot Paolo
Reviewed-by: Eric Dumazet edumazet@google.com
On 11/5/24 11:23 AM, Paolo Abeni wrote:
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at dst_init+0x84/0x4a0 dst_alloc+0x97/0x150 ip6_dst_alloc+0x23/0x90 ip6_rt_pcpu_alloc+0x1e6/0x520 ip6_pol_route+0x56f/0x840 fib6_rule_lookup+0x334/0x630 ip6_route_output_flags+0x259/0x480 ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 ip6_dst_lookup_flow+0x88/0x190 udp_tunnel6_dst_lookup+0x2a7/0x4c0 vxlan_xmit_one+0xbde/0x4a50 [vxlan] vxlan_xmit+0x9ad/0xf20 [vxlan] dev_hard_start_xmit+0x10e/0x360 __dev_queue_xmit+0xf95/0x18c0 arp_solicit+0x4a2/0xe00 neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:
./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
are you sure that is the correct Fixes? That commit is June 2019 and there have been stable periods since then without netdev release problems.
Signed-off-by: Paolo Abeni pabeni@redhat.com
v1 -> v2:
- dropped unintended whitespace change
net/ipv6/route.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Reviewed-by: David Ahern dsahern@kernel.org
On 11/5/24 22:40, David Ahern wrote:
On 11/5/24 11:23 AM, Paolo Abeni wrote:
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test:
unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at dst_init+0x84/0x4a0 dst_alloc+0x97/0x150 ip6_dst_alloc+0x23/0x90 ip6_rt_pcpu_alloc+0x1e6/0x520 ip6_pol_route+0x56f/0x840 fib6_rule_lookup+0x334/0x630 ip6_route_output_flags+0x259/0x480 ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 ip6_dst_lookup_flow+0x88/0x190 udp_tunnel6_dst_lookup+0x2a7/0x4c0 vxlan_xmit_one+0xbde/0x4a50 [vxlan] vxlan_xmit+0x9ad/0xf20 [vxlan] dev_hard_start_xmit+0x10e/0x360 __dev_queue_xmit+0xf95/0x18c0 arp_solicit+0x4a2/0xe00 neigh_probe+0xaa/0xf0
While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info.
Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown().
Tested running:
./pmtu.sh cleanup_ipv6_exception
in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations.
Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
are you sure that is the correct Fixes? That commit is June 2019 and there have been stable periods since then without netdev release problems.
"Sure" is a big word ;) AFAICS the mentioned commit let fib6_info store indirectly the extra dev reference via nexthop and does not clean it at device removal time.
Note that the issue is not deterministic - I needed ~30 mptu.sh iterations in a row to see it, so it could go unnoticed for a long time.
Signed-off-by: Paolo Abeni pabeni@redhat.com
v1 -> v2:
- dropped unintended whitespace change
net/ipv6/route.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Reviewed-by: David Ahern dsahern@kernel.org
Thanks!
Paolo
A recent refactor transformed the check for process completion in a true statement, due to a typo.
As a result, the relevant test-case is unable to catch the regression it was supposed to detect.
Restore the correct condition.
Fixes: 691bb4e49c98 ("selftests: net: avoid just another constant wait") Signed-off-by: Paolo Abeni pabeni@redhat.com --- tools/testing/selftests/net/pmtu.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh index 569bce8b6383..6c651c880fe8 100755 --- a/tools/testing/selftests/net/pmtu.sh +++ b/tools/testing/selftests/net/pmtu.sh @@ -2056,7 +2056,7 @@ check_running() { pid=${1} cmd=${2}
- [ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "{cmd}" ] + [ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "${cmd}" ] }
test_cleanup_vxlanX_exception() {
On 11/5/24 11:23 AM, Paolo Abeni wrote:
A recent refactor transformed the check for process completion in a true statement, due to a typo.
As a result, the relevant test-case is unable to catch the regression it was supposed to detect.
Restore the correct condition.
Fixes: 691bb4e49c98 ("selftests: net: avoid just another constant wait") Signed-off-by: Paolo Abeni pabeni@redhat.com
tools/testing/selftests/net/pmtu.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by: David Ahern dsahern@kernel.org
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski kuba@kernel.org:
On Tue, 5 Nov 2024 19:23:49 +0100 you wrote:
This addresses the infamous unregister_netdevice splat in net selftests; the actual fix is carried by the first patch, while the 2nd one addresses a related problem in the relevant test that was patially hiding the problem.
Targeting net-next as the issue is quite old and I feel a little lost in the fib info/nh jungle.
[...]
Here is the summary with links: - [v2,net-next,1/2] ipv6: release nexthop on device removal https://git.kernel.org/netdev/net-next/c/eb02688c5c45 - [v2,net-next,2/2] selftests: net: really check for bg process completion https://git.kernel.org/netdev/net-next/c/52ed077aa633
You are awesome, thank you!
linux-kselftest-mirror@lists.linaro.org
-
David Ahern -
Eric Dumazet -
Paolo Abeni -
patchwork-bot+netdevbpf＠kernel.org