On Tue, 2019-07-02 at 10:02 +0200, Greg Kroah-Hartman wrote:
From: Xin Long lucien.xin@gmail.com
commit c3bcde026684c62d7a2b6f626dc7cf763833875c upstream.
udp_tunnel(6)_xmit_skb() called by tipc_udp_xmit() expects a tunnel device to count packets on dev->tstats, a perpcu variable. However, TIPC is using udp tunnel with no tunnel device, and pass the lower dev, like veth device that only initializes dev->lstats(a perpcu variable) when creating it.
Hi,
This tipc patch added in 4.14.132 is triggering a crash for me, revert fixes it.
Anyone have ideas if some other commits missing in 4.14.x to make this work...?
# modprobe tipc # tipc node set addr 1.1.2 # tipc bearer enable media udp name UDP1 localip 192.168.1.15
[ 143.105529] Own node address <1.1.2>, network identity 4711 [ 172.087098] BUG: unable to handle kernel NULL pointer dereference at 00000000000004f0 [ 172.088375] IP: iptunnel_xmit+0x15e/0x1e0 [ 172.089072] PGD 8000000231306067 P4D 8000000231306067 PUD 2356e1067 PMD 0 [ 172.090094] Oops: 0000 [#1] SMP PTI [ 172.090610] Modules linked in: tipc ip6_udp_tunnel udp_tunnel isofs kvm_intel kvm irqbypass sch_fq_codel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ata_piix dm_mirror dm_region_hash dm_log dm_mod dax autofs4 [ 172.093293] CPU: 1 PID: 747 Comm: tipc Not tainted 4.14.134-1.x86_64 #1 [ 172.094448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-2.fc30 04/01/2014 [ 172.095703] task: ffff8b99f12c0000 task.stack: ffff9ab481198000 [ 172.096731] RIP: 0010:iptunnel_xmit+0x15e/0x1e0 [ 172.097460] RSP: 0018:ffff9ab48119ba00 EFLAGS: 00010202 [ 172.098214] RAX: 0000000000000000 RBX: ffffffffbf4d8140 RCX: 000000000000008c [ 172.099320] RDX: 0000000000000001 RSI: 00000000fffffe01 RDI: ffffffffbe944d62 [ 172.100392] RBP: ffff8b99f1e7ed00 R08: ffff8b99ffc64520 R09: 0000000000000000 [ 172.101451] R10: 000000023426d000 R11: 0000000000000002 R12: 0000000000000000 [ 172.102607] R13: 0000000000000040 R14: 0000000000000000 R15: ffff8b99f426e0e8 [ 172.103728] FS: 00007efc82b96800(0000) GS:ffff8b99ffc40000(0000) knlGS:0000000000000000 [ 172.104976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 172.105821] CR2: 00000000000004f0 CR3: 0000000234250001 CR4: 00000000003606e0 [ 172.106981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 172.108120] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 172.109386] Call Trace: [ 172.109808] tipc_udp_xmit.isra.18+0x1a7/0x1c0 [tipc] [ 172.110687] ? __internal_add_timer+0x1a/0x50 [ 172.111369] ? __skb_clone+0x29/0x130 [ 172.111999] tipc_bearer_xmit_skb+0x4d/0x80 [tipc] [ 172.112845] tipc_enable_bearer+0x2b9/0x3c0 [tipc] [ 172.113637] ? __nla_put+0xc/0x20 [ 172.114213] tipc_nl_bearer_enable+0xca/0x100 [tipc] [ 172.114952] genl_family_rcv_msg+0x190/0x390 [ 172.115748] genl_rcv_msg+0x47/0x90 [ 172.116287] ? __alloc_skb+0x72/0x1b0 [ 172.116898] ? genl_family_rcv_msg+0x390/0x390 [ 172.117669] netlink_rcv_skb+0x3d/0x100 [ 172.118361] genl_rcv+0x24/0x40 [ 172.119005] netlink_unicast+0x16d/0x230 [ 172.119777] netlink_sendmsg+0x1ae/0x3c0 [ 172.120525] SYSC_sendto+0xe6/0x140 [ 172.121248] ? SYSC_getsockname+0x81/0xa0 [ 172.121989] ? sock_alloc_file+0x97/0x120 [ 172.122645] ? sock_map_fd+0x3d/0x60 [ 172.123278] do_syscall_64+0x74/0x190 [ 172.123911] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 172.124716] RIP: 0033:0x7efc82d6ac6b [ 172.125368] RSP: 002b:00007fff40411ae8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 172.126486] RAX: ffffffffffffffda RBX: 0000000001dfca20 RCX: 00007efc82d6ac6b [ 172.127632] RDX: 0000000000000054 RSI: 00007fff40411b60 RDI: 0000000000000003 [ 172.128765] RBP: 00007fff40411b50 R08: 00007efc82e36000 R09: 000000000000000c [ 172.129793] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff40411b60 [ 172.130799] R13: 00007fff40412d10 R14: 000000000040bb44 R15: 0000000000000000 [ 172.131868] Code: 01 00 00 00 85 d2 0f 44 d0 e8 1f f3 fa ff 48 8b 74 24 08 4c 89 fa 48 89 df e8 9f 94 fb ff 83 e0 fd 75 35 8b 4c 24 1c 85 c9 7e 2b <49> 8b 84 24 f0 04 00 00 65 48 03 05 aa 29 68 41 48 83 40 10 01 [ 172.134773] RIP: iptunnel_xmit+0x15e/0x1e0 RSP: ffff9ab48119ba00 [ 172.135697] CR2: 00000000000004f0 [ 172.136305] ---[ end trace 27f7522ade26797f ]---
Later iptunnel_xmit_stats() called by ip(6)tunnel_xmit() thinks the dev as a tunnel device, and uses dev->tstats instead of dev->lstats. tstats' each pointer points to a bigger struct than lstats, so when tstats-
tx_bytes is
increased, other percpu variable's members could be overwritten.
syzbot has reported quite a few crashes due to fib_nh_common percpu member 'nhc_pcpu_rth_output' overwritten, call traces are like:
BUG: KASAN: slab-out-of-bounds in rt_cache_valid+0x158/0x190 net/ipv4/route.c:1556 rt_cache_valid+0x158/0x190 net/ipv4/route.c:1556 __mkroute_output net/ipv4/route.c:2332 [inline] ip_route_output_key_hash_rcu+0x819/0x2d50 net/ipv4/route.c:2564 ip_route_output_key_hash+0x1ef/0x360 net/ipv4/route.c:2393 __ip_route_output_key include/net/route.h:125 [inline] ip_route_output_flow+0x28/0xc0 net/ipv4/route.c:2651 ip_route_output_key include/net/route.h:135 [inline] ...
or:
kasan: GPF could be caused by NULL-ptr deref or user memory access RIP: 0010:dst_dev_put+0x24/0x290 net/core/dst.c:168 <IRQ> rt_fibinfo_free_cpus net/ipv4/fib_semantics.c:200 [inline] free_fib_info_rcu+0x2e1/0x490 net/ipv4/fib_semantics.c:217 __rcu_reclaim kernel/rcu/rcu.h:240 [inline] rcu_do_batch kernel/rcu/tree.c:2437 [inline] invoke_rcu_callbacks kernel/rcu/tree.c:2716 [inline] rcu_process_callbacks+0x100a/0x1ac0 kernel/rcu/tree.c:2697 ...
The issue exists since tunnel stats update is moved to iptunnel_xmit by Commit 039f50629b7f ("ip_tunnel: Move stats update to iptunnel_xmit()"), and here to fix it by passing a NULL tunnel dev to udp_tunnel(6)_xmit_skb so that the packets counting won't happen on dev->tstats.
Reported-by: syzbot+9d4c12bfd45a58738d0a@syzkaller.appspotmail.com Reported-by: syzbot+a9e23ea2aa21044c2798@syzkaller.appspotmail.com Reported-by: syzbot+c4c4b2bb358bb936ad7e@syzkaller.appspotmail.com Reported-by: syzbot+0290d2290a607e035ba1@syzkaller.appspotmail.com Reported-by: syzbot+a43d8d4e7e8a7a9e149e@syzkaller.appspotmail.com Reported-by: syzbot+a47c5f4c6c00fc1ed16e@syzkaller.appspotmail.com Fixes: 039f50629b7f ("ip_tunnel: Move stats update to iptunnel_xmit()") Signed-off-by: Xin Long lucien.xin@gmail.com Signed-off-by: David S. Miller davem@davemloft.net Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org
net/tipc/udp_media.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
--- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -174,7 +174,6 @@ static int tipc_udp_xmit(struct net *net goto tx_error; }
skb->dev = rt->dst.dev;ipv4.s_addr,
dst->ipv4.s_addr, 0, ttl, 0, src-port,
@@ -193,10 +192,9 @@ static int tipc_udp_xmit(struct net *net if (err) goto tx_error; ttl = ip6_dst_hoplimit(ndst);
err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb,ndst->dev, &src->ipv6,&dst->ipv6, 0, ttl, 0, src-port,
dst->port, false);
err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb,NULL,
&src->ipv6, &dst->ipv6, 0,ttl, 0,
src->port, dst->port,false); #endif } return err;