For now, the tcp socket lookup will terminate if the socket is reuse port in inet_lhash2_lookup(), which makes the socket is not the best match.
For example, we have socket1 and socket2 both listen on "0.0.0.0:1234", but socket1 bind on "eth0". We create socket1 first, and then socket2. Then, all connections will goto socket2, which is not expected, as socket1 has higher priority.
The 1st patch fix this problem, and the 2nd patch is a selftests for this problem. Without the 1st patch, the selftests will fail with:
$ ./tcp_reuseport.py TAP version 13 1..1 FAIL: wrong assignment not ok 1 tcp_reuseport.test_reuseport_select Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
With the 1st patch, it will success: $ ./tcp_reuseport.py TAP version 13 1..1 SUCCESS: assigned properly: (<socket.socket fd=6, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43140)>, ('127.0.0.1', 43140)) SUCCESS: assigned properly: (<socket.socket fd=5, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43146)>, ('127.0.0.1', 43146)) SUCCESS: assigned properly: (<socket.socket fd=6, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43162)>, ('127.0.0.1', 43162)) ok 1 tcp_reuseport.test_reuseport_select Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
Changes since V2: * use the approach in V1 * add the Fixes tag in the 1st patch * introduce the selftests
Menglong Dong (2): net: tcp: lookup the best matched listen socket selftests/net: test TCP reuseport socket selection
net/ipv4/inet_hashtables.c | 13 +++---- net/ipv6/inet6_hashtables.c | 13 +++---- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++ 4 files changed, 51 insertions(+), 12 deletions(-) create mode 100755 tools/testing/selftests/net/tcp_reuseport.py
For now, the tcp socket lookup will terminate if the socket is reuse port in inet_lhash2_lookup(), which makes the socket is not the best match.
For example, we have socket1 and socket2 both listen on "0.0.0.0:1234", but socket1 bind on "eth0". We create socket1 first, and then socket2. Then, all connections will goto socket2, which is not expected, as socket1 has higher priority.
This can cause unexpected behavior if TCP MD5 keys is used, as described in Documentation/networking/vrf.rst -> Applications.
Therefor, we lookup the best matched socket first, and then do the reuse port logic. This can increase some overhead if there are many reuse port socket :/
Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection") Signed-off-by: Menglong Dong dongml2@chinatelecom.cn --- v3: * use the approach in V1 * add the Fixes tag --- net/ipv4/inet_hashtables.c | 13 +++++++------ net/ipv6/inet6_hashtables.c | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index ceeeec9b7290..51751337f394 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net, sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) { score = compute_score(sk, net, hnum, daddr, dif, sdif); if (score > hiscore) { - result = inet_lookup_reuseport(net, sk, skb, doff, - saddr, sport, daddr, hnum, inet_ehashfn); - if (result) - return result; - result = sk; hiscore = score; } }
- return result; + if (!result) + return NULL; + + sk = inet_lookup_reuseport(net, result, skb, doff, + saddr, sport, daddr, hnum, inet_ehashfn); + + return sk ? sk : result; }
struct sock *inet_lookup_run_sk_lookup(const struct net *net, diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index 76ee521189eb..2554f26d6c20 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -161,17 +161,18 @@ static struct sock *inet6_lhash2_lookup(const struct net *net, sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) { score = compute_score(sk, net, hnum, daddr, dif, sdif); if (score > hiscore) { - result = inet6_lookup_reuseport(net, sk, skb, doff, - saddr, sport, daddr, hnum, inet6_ehashfn); - if (result) - return result; - result = sk; hiscore = score; } }
- return result; + if (!result) + return NULL; + + sk = inet6_lookup_reuseport(net, result, skb, doff, + saddr, sport, daddr, hnum, inet6_ehashfn); + + return sk ? sk : result; }
struct sock *inet6_lookup_run_sk_lookup(const struct net *net,
Hi Menglong,
On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong menglong8.dong@gmail.com wrote:
For now, the tcp socket lookup will terminate if the socket is reuse port in inet_lhash2_lookup(), which makes the socket is not the best match.
For example, we have socket1 and socket2 both listen on "0.0.0.0:1234", but socket1 bind on "eth0". We create socket1 first, and then socket2. Then, all connections will goto socket2, which is not expected, as socket1 has higher priority.
This can cause unexpected behavior if TCP MD5 keys is used, as described in Documentation/networking/vrf.rst -> Applications.
Therefor, we lookup the best matched socket first, and then do the reuse
s/Therefor/Therefore
port logic. This can increase some overhead if there are many reuse port socket :/
Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection") Signed-off-by: Menglong Dong dongml2@chinatelecom.cn
v3:
- use the approach in V1
- add the Fixes tag
net/ipv4/inet_hashtables.c | 13 +++++++------ net/ipv6/inet6_hashtables.c | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index ceeeec9b7290..51751337f394 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net, sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) { score = compute_score(sk, net, hnum, daddr, dif, sdif); if (score > hiscore) {
result = inet_lookup_reuseport(net, sk, skb, doff,saddr, sport, daddr, hnum, inet_ehashfn);if (result)return result;result = sk; hiscore = score; } }return result;
if (!result)return NULL;sk = inet_lookup_reuseport(net, result, skb, doff,saddr, sport, daddr, hnum, inet_ehashfn);return sk ? sk : result;}
IMHO, I don't see it as a bugfix. So can you elaborate on what the exact side effect you're faced with is when the algorithm finally prefers socket2 (without this patch)?
AFAIK, the current approach breaks the initial design and might make the whole lookup process take a longer time in certain cases like you mentioned.
Thanks, Jason
On Sat, Aug 2, 2025 at 9:06 PM Jason Xing kerneljasonxing@gmail.com wrote:
Hi Menglong,
On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong menglong8.dong@gmail.com wrote:
For now, the tcp socket lookup will terminate if the socket is reuse port in inet_lhash2_lookup(), which makes the socket is not the best match.
For example, we have socket1 and socket2 both listen on "0.0.0.0:1234", but socket1 bind on "eth0". We create socket1 first, and then socket2. Then, all connections will goto socket2, which is not expected, as socket1 has higher priority.
This can cause unexpected behavior if TCP MD5 keys is used, as described in Documentation/networking/vrf.rst -> Applications.
Therefor, we lookup the best matched socket first, and then do the reuse
s/Therefor/Therefore
port logic. This can increase some overhead if there are many reuse port socket :/
Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection") Signed-off-by: Menglong Dong dongml2@chinatelecom.cn
v3:
- use the approach in V1
- add the Fixes tag
net/ipv4/inet_hashtables.c | 13 +++++++------ net/ipv6/inet6_hashtables.c | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index ceeeec9b7290..51751337f394 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net, sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) { score = compute_score(sk, net, hnum, daddr, dif, sdif); if (score > hiscore) {
result = inet_lookup_reuseport(net, sk, skb, doff,saddr, sport, daddr, hnum, inet_ehashfn);if (result)return result;result = sk; hiscore = score; } }return result;
if (!result)return NULL;sk = inet_lookup_reuseport(net, result, skb, doff,saddr, sport, daddr, hnum, inet_ehashfn);return sk ? sk : result;}
IMHO, I don't see it as a bugfix. So can you elaborate on what the exact side effect you're faced with is when the algorithm finally prefers socket2 (without this patch)?
Hi, Jason. The case is that the user has several NIC, and there are some sockets that are binded to them, who listen on TCP port 6666. And a global socket doesn't bind any NIC and listens on TCP port 6666.
In theory, the connection request from the NIC will goto the listen socket that is binded on it. When on socket is binded on the NIC, it goto the global socket. However, the connection request always goto the global socket, which is not expected.
What's worse is that when TCP MD5 is used on the socket, the connection will fail :/
More discussion can be found here: https://lore.kernel.org/netdev/20250801090949.129941-1-dongml2@chinatelecom.... https://lore.kernel.org/netdev/20250731123309.184496-1-dongml2@chinatelecom....
AFAIK, the current approach breaks the initial design and might make the whole lookup process take a longer time in certain cases like you mentioned.
Kuniyuki said he will post some patch when the net-next is open, which reorder the socket in __inet_hash() to make the reuseport socket lookup keep O(1).
So this patch is more like a temporary solution :/
Thanks! Menglong Dong
Thanks, Jason
The test script is provided by Kuniyuki in [1], which is used to test the selection of the TCP reuseport socket problem.
Link: https://lore.kernel.org/netdev/20250801040757.1599996-1-kuniyu@google.com/ [1] Signed-off-by: Menglong Dong dongml2@chinatelecom.cn --- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 tools/testing/selftests/net/tcp_reuseport.py
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile index b31a71f2b372..0f4c3eea9709 100644 --- a/tools/testing/selftests/net/Makefile +++ b/tools/testing/selftests/net/Makefile @@ -117,6 +117,7 @@ TEST_GEN_FILES += tfo TEST_PROGS += tfo_passive.sh TEST_PROGS += broadcast_pmtu.sh TEST_PROGS += ipv6_force_forwarding.sh +TEST_PROGS += tcp_reuseport.py
# YNL files, must be before "include ..lib.mk" YNL_GEN_FILES := busy_poller netlink-dumps diff --git a/tools/testing/selftests/net/tcp_reuseport.py b/tools/testing/selftests/net/tcp_reuseport.py new file mode 100755 index 000000000000..eaeb7096382e --- /dev/null +++ b/tools/testing/selftests/net/tcp_reuseport.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import os + +from lib.py import ksft_run, ksft_exit +from socket import * + +def test_reuseport_select() -> None: + s1 = socket() + s1.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1) + s1.setsockopt(SOL_SOCKET, SO_BINDTODEVICE, b'lo') + s1.listen() + s1.setblocking(False) + + s2 = socket() + s2.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1) + s2.bind(s1.getsockname()) + s2.listen() + s2.setblocking(False) + + for i in range(3): + c = socket() + c.connect(s1.getsockname()) + try: + print("SUCCESS: assigned properly:", s1.accept()) + except: + print("FAIL: wrong assignment") + os.sys.exit(1) + +def main() -> None: + ksft_run([test_reuseport_select]) + ksft_exit() + +if __name__ == "__main__": + main()
linux-kselftest-mirror@lists.linaro.org
-
Jason Xing -
Menglong Dong