Abstract === This patchset improves the performance of sockmap by providing CPU affinity, resulting in a 1-10x increase in throughput.
Motivation === Traditional user-space reverse proxy:
Reserve Proxy _________________ client -> | fd1 <-> fd2 | -> server |_________________|
Using sockmap for reverse proxy:
Reserve Proxy _________________ client -> | fd1 <-> fd2 | -> server | |_________________| | | | | | | _________ | | | sockmap | | --> |_________| -->
By adding fds to sockmap and using a BPF program, we can quickly forward data and avoid data copying between user space and kernel space.
Mainstream multi-process reverse proxy applications, such as Nginx and HAProxy, support CPU affinity settings, which allow each process to be pinned to a specific CPU, avoiding conflicts between data plane processes and other processes, especially in multi-tenant environments.
Current Issues === The current design of sockmap uses a workqueue to forward ingress_skb and wakes up the workqueue without specifying a CPU (by calling schedule_delayed_work()). In the current implementation of schedule_delayed_work, it tends to run the workqueue on the current CPU.
This approach has a high probability of running on the current CPU, which is the same CPU that handles the net rx soft interrupt, especially for programs that access each other using local interfaces.
The loopback driver's transmit interface, loopback_xmit(), directly calls __netif_rx() on the current CPU, which means that the CPU handling sockmap's workqueue and the client's sending CPU are the same, resulting in contention.
For a TCP flow, if the request or response is very large, the psock->ingress_skb queue can become very long. When the workqueue traverses this queue to forward the data, it can consume a significant amount of CPU time.
Solution === Configuring RPS on a loopback interface can be useful, but it will trigger additional softirq, and furthermore, it fails to achieve our expected effect of CPU isolation from other processes.
Instead, we provide a kfunc that allow users to specify the CPU on which the workqueue runs through a BPF program.
We can use the existing benchmark to test the performance, which allows us to evaluate the effectiveness of this optimization.
Because we use local interfaces for communication and the client consumes a significant amount of CPU when sending data, this prevents the workqueue from processing ingress_skb in a timely manner, ultimately causing the server to fail to read data quickly.
Without cpu-affinity: ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 36.031us): Send Speed 1143.693 MB/s ... Rcv Speed 109.572 MB/s Iter 1 ( 0.608us): Send Speed 1320.550 MB/s ... Rcv Speed 48.103 MB/s Iter 2 ( -5.448us): Send Speed 1314.790 MB/s ... Rcv Speed 47.842 MB/s Iter 3 ( -0.613us): Send Speed 1320.158 MB/s ... Rcv Speed 46.531 MB/s Iter 4 ( -3.441us): Send Speed 1319.375 MB/s ... Rcv Speed 46.662 MB/s Iter 5 ( 3.764us): Send Speed 1166.667 MB/s ... Rcv Speed 42.467 MB/s Iter 6 ( -4.404us): Send Speed 1319.508 MB/s ... Rcv Speed 47.973 MB/s Summary: total trans 7758 MB ± 1293.506 MB/s
Without cpu-affinity(RPS enabled): ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 28.925us): Send Speed 1630.357 MB/s ... Rcv Speed 850.960 MB/s Iter 1 ( -2.042us): Send Speed 1644.564 MB/s ... Rcv Speed 822.478 MB/s Iter 2 ( 0.754us): Send Speed 1644.297 MB/s ... Rcv Speed 850.787 MB/s Iter 3 ( 0.159us): Send Speed 1644.429 MB/s ... Rcv Speed 850.198 MB/s Iter 4 ( -2.898us): Send Speed 1646.924 MB/s ... Rcv Speed 830.867 MB/s Iter 5 ( -0.210us): Send Speed 1649.410 MB/s ... Rcv Speed 824.246 MB/s Iter 6 ( -1.448us): Send Speed 1650.723 MB/s ... Rcv Speed 808.256 MB/s
With cpu-affinity(RPS disabled): ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify --cpu-affinity Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 36.051us): Send Speed 1883.437 MB/s ... Rcv Speed 1865.087 MB/s Iter 1 ( 1.246us): Send Speed 1900.542 MB/s ... Rcv Speed 1761.737 MB/s Iter 2 ( -8.595us): Send Speed 1883.128 MB/s ... Rcv Speed 1860.714 MB/s Iter 3 ( 7.033us): Send Speed 1890.831 MB/s ... Rcv Speed 1806.684 MB/s Iter 4 ( -8.397us): Send Speed 1884.700 MB/s ... Rcv Speed 1973.568 MB/s Iter 5 ( -1.822us): Send Speed 1894.125 MB/s ... Rcv Speed 1775.046 MB/s Iter 6 ( 4.936us): Send Speed 1877.597 MB/s ... Rcv Speed 1959.320 MB/s Summary: total trans 11328 MB ± 1888.507 MB/s
Appendix === Through this optimization, we discovered that sk_mem_charge() and sk_mem_uncharge() have concurrency issues. The performance improvement brought by this optimization has made these concurrency issues more evident.
This concurrency issue can cause the WARN_ON_ONCE(sk->sk_forward_alloc) check to be triggered when the socket is released. Since this patch is a feature-type patch and does not intend to fix this bug, I will provide additional patches to fix this issue later.
Jiayuan Chen (3): bpf, sockmap: Introduce a new kfunc for sockmap bpf, sockmap: Affinitize workqueue to a specific CPU selftest/bpf/benchs: Add cpu-affinity for sockmap bench
Documentation/bpf/map_sockmap.rst | 14 +++++++ include/linux/skmsg.h | 15 +++++++ kernel/bpf/btf.c | 3 ++ net/core/skmsg.c | 10 +++-- net/core/sock_map.c | 39 +++++++++++++++++++ .../selftests/bpf/benchs/bench_sockmap.c | 35 +++++++++++++++-- tools/testing/selftests/bpf/bpf_kfuncs.h | 6 +++ .../selftests/bpf/progs/bench_sockmap_prog.c | 7 ++++ 8 files changed, 122 insertions(+), 7 deletions(-)
Since the helper list is effectively frozen and the existing helpers cannot be extended, we add a new kfunc instead which simply set the redir_cpu to psock.
The new kfunc is used to set redir_cpu to psock.
All these changes conform to the kfuncs.rst documentation.
Signed-off-by: Jiayuan Chen jiayuan.chen@linux.dev --- Documentation/bpf/map_sockmap.rst | 14 +++++++++++ include/linux/skmsg.h | 3 +++ kernel/bpf/btf.c | 3 +++ net/core/skmsg.c | 1 + net/core/sock_map.c | 39 +++++++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)
diff --git a/Documentation/bpf/map_sockmap.rst b/Documentation/bpf/map_sockmap.rst index 2d630686a00b..eca3dfc1c85f 100644 --- a/Documentation/bpf/map_sockmap.rst +++ b/Documentation/bpf/map_sockmap.rst @@ -212,6 +212,20 @@ following cases:
Returns 0
+bpf_sk_skb_set_redirect_cpu() +^^^^^^^^^^^^^^^^^^^^^^ +.. code-block:: c + + int bpf_sk_skb_set_redirect_cpu(struct __sk_buff *s, int redir_cpu) + +This kfunc ``bpf_sk_skb_set_redirect_cpu()`` is available to +``BPF_PROG_TYPE_SK_SKB`` BPF programs. It sets the CPU affinity, allowing the +sockmap packet redirecting process to run on the specified CPU as much as +possible, helping users reduce the interference between the sockmap redirecting +background thread and other threads. + +Returns 0 on success, or a negative error in case of failure. + bpf_msg_cork_bytes() ^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index 0b9095a281b8..b888481a845d 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -16,6 +16,8 @@ #define MAX_MSG_FRAGS MAX_SKB_FRAGS #define NR_MSG_FRAG_IDS (MAX_MSG_FRAGS + 1)
+#define BPF_SK_REDIR_CPU_UNSET -1 + enum __sk_action { __SK_DROP = 0, __SK_PASS, @@ -86,6 +88,7 @@ struct sk_psock { u32 apply_bytes; u32 cork_bytes; u32 eval; + s32 redir_cpu; bool redir_ingress; /* undefined if sk_redir is null */ struct sk_msg *cork; struct sk_psock_progs progs; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index a91822bae043..2a8f59e2c639 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -219,6 +219,7 @@ enum btf_kfunc_hook { BTF_KFUNC_HOOK_LWT, BTF_KFUNC_HOOK_NETFILTER, BTF_KFUNC_HOOK_KPROBE, + BTF_KFUNC_HOOK_SK_MSG, BTF_KFUNC_HOOK_MAX, };
@@ -8649,6 +8650,8 @@ static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type) return BTF_KFUNC_HOOK_SCHED_ACT; case BPF_PROG_TYPE_SK_SKB: return BTF_KFUNC_HOOK_SK_SKB; + case BPF_PROG_TYPE_SK_MSG: + return BTF_KFUNC_HOOK_SK_MSG; case BPF_PROG_TYPE_SOCKET_FILTER: return BTF_KFUNC_HOOK_SOCKET_FILTER; case BPF_PROG_TYPE_LWT_OUT: diff --git a/net/core/skmsg.c b/net/core/skmsg.c index 276934673066..292752c783b5 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -741,6 +741,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node) psock->saved_destroy = prot->destroy; psock->saved_close = prot->close; psock->saved_write_space = sk->sk_write_space; + psock->redir_cpu = BPF_SK_REDIR_CPU_UNSET;
INIT_LIST_HEAD(&psock->link); spin_lock_init(&psock->link_lock); diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 82a14f131d00..9f1e531a3807 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -701,6 +701,45 @@ const struct bpf_func_proto bpf_msg_redirect_map_proto = { .arg4_type = ARG_ANYTHING, };
+__bpf_kfunc_start_defs(); + +__bpf_kfunc int bpf_sk_skb_set_redirect_cpu(struct __sk_buff *s, int redir_cpu) +{ + struct sk_buff *skb = (struct sk_buff *)s; + struct sock *sk = skb->sk; + struct sk_psock *psock; + + WARN_ON_ONCE(!rcu_read_lock_held()); + + if (!sk || redir_cpu >= num_possible_cpus()) + return -EINVAL; + + psock = sk_psock(sk); + if (!psock) + return -ENOENT; + + psock->redir_cpu = redir_cpu; + return 0; +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_sk_sockmap_kfunc_ids) +BTF_ID_FLAGS(func, bpf_sk_skb_set_redirect_cpu) +BTF_KFUNCS_END(bpf_sk_sockmap_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_sk_sockmap_kfunc_set = { + .owner = THIS_MODULE, + .set = &bpf_sk_sockmap_kfunc_ids, +}; + +static int init_sockmap_subsystem(void) +{ + return register_btf_kfunc_id_set(BPF_PROG_TYPE_SK_SKB, &bpf_sk_sockmap_kfunc_set); +} + +late_initcall(init_sockmap_subsystem); + struct sock_map_seq_info { struct bpf_map *map; struct sock *sk;
On Mon, Apr 28, 2025 at 04:16:52PM +0800, Jiayuan Chen wrote:
+bpf_sk_skb_set_redirect_cpu() +^^^^^^^^^^^^^^^^^^^^^^ +.. code-block:: c
- int bpf_sk_skb_set_redirect_cpu(struct __sk_buff *s, int redir_cpu)
+This kfunc ``bpf_sk_skb_set_redirect_cpu()`` is available to +``BPF_PROG_TYPE_SK_SKB`` BPF programs. It sets the CPU affinity, allowing the +sockmap packet redirecting process to run on the specified CPU as much as +possible, helping users reduce the interference between the sockmap redirecting +background thread and other threads.
I am wondering if it is a better idea to use BPF_MAP_TYPE_CPUMAP for redirection here instead? Like we did for bpf_redirect_map(). At least we would not need to store CPU in psock with this approach.
Thanks.
2025/4/29 08:56, "Cong Wang" xiyou.wangcong@gmail.com wrote:
On Mon, Apr 28, 2025 at 04:16:52PM +0800, Jiayuan Chen wrote:
+bpf_sk_skb_set_redirect_cpu()
+^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: c
int bpf_sk_skb_set_redirect_cpu(struct __sk_buff *s, int redir_cpu)
+This kfunc ``bpf_sk_skb_set_redirect_cpu()`` is available to
+``BPF_PROG_TYPE_SK_SKB`` BPF programs. It sets the CPU affinity, allowing the
+sockmap packet redirecting process to run on the specified CPU as much as
+possible, helping users reduce the interference between the sockmap redirecting
+background thread and other threads.
I am wondering if it is a better idea to use BPF_MAP_TYPE_CPUMAP for
redirection here instead? Like we did for bpf_redirect_map(). At least
we would not need to store CPU in psock with this approach.
Thanks.
You mean to use BPF_MAP_TYPE_CPUMAP with XDP to redirect packets to a specific CPU?
I tested and found such overhead: 1、Needing to parse the L4 header from the L2 header to obtain the 5-tuple, and then maintaining an additional map to store the relationship between each five-tuple and process/CPU. Compared to multi-process scenario, with one process binding to one CPU and one map, I can directly use a global variable to let the BPF program know which thread it should use, especially for programs that enable reuseport.
2、Furthermore, regarding performance, I tested with cpumap and the results were lower than expected. This is because loopback only has xdp_generic mode and the problem I described in cover letter is actually occurred on loopback...
Code: ''' struct { __uint(type, BPF_MAP_TYPE_CPUMAP); __uint(key_size, sizeof(__u32)); __uint(value_size, sizeof(struct bpf_cpumap_val)); __uint(max_entries, 64); } cpu_map SEC(".maps");
SEC("xdp") int xdp_stats1_func(struct xdp_md *ctx) { /* Real world: * 1. get 5-tuple from ctx * 2. get corresponding cpu of current skb through XX_MAP */ int ret = bpf_redirect_map(&cpu_map, 3, 0); // redirct to 3 return ret; } '''
Result: ''' ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 36.439us): Send Speed 561.496 MB/s ... Rcv Speed 33.264 MB/s Iter 1 ( -7.448us): Send Speed 558.443 MB/s ... Rcv Speed 32.611 MB/s Iter 2 ( -2.245us): Send Speed 557.131 MB/s ... Rcv Speed 33.004 MB/s Iter 3 ( -2.845us): Send Speed 547.374 MB/s ... Rcv Speed 33.331 MB/s Iter 4 ( 0.745us): Send Speed 562.891 MB/s ... Rcv Speed 34.117 MB/s Iter 5 ( -2.056us): Send Speed 560.994 MB/s ... Rcv Speed 33.069 MB/s Iter 6 ( 5.343us): Send Speed 562.038 MB/s ... Rcv Speed 33.200 MB/s '''
Instead, we can introduce a new kfunc to specify the CPU used by the backlog running thread, which can avoid using XDP. After all, this is a "problem" brought by the BPF L7 framework itself, and it's better to solve it ourselves.
Introduce a sk_psock_schedule_delayed_work() wrapper function, which calls schedule_delayed_work_on() to specify the CPU for running the workqueue if the BPF program has set the redirect CPU using bpf_sk_skb_set_redirect_cpu(). Otherwise, it falls back to the original logic.
Signed-off-by: Jiayuan Chen jiayuan.chen@linux.dev --- include/linux/skmsg.h | 12 ++++++++++++ net/core/skmsg.c | 9 +++++---- 2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index b888481a845d..21c7dd47186f 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -396,6 +396,18 @@ static inline void sk_psock_report_error(struct sk_psock *psock, int err) sk_error_report(sk); }
+static inline void sk_psock_schedule_delayed_work(struct sk_psock *psock, + int delay) +{ + s32 redir_cpu = psock->redir_cpu; + + if (redir_cpu != BPF_SK_REDIR_CPU_UNSET) + schedule_delayed_work_on(redir_cpu, &psock->work, + delay); + else + schedule_delayed_work(&psock->work, delay); +} + struct sk_psock *sk_psock_init(struct sock *sk, int node); void sk_psock_stop(struct sk_psock *psock);
diff --git a/net/core/skmsg.c b/net/core/skmsg.c index 292752c783b5..af00c09263a8 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -689,7 +689,7 @@ static void sk_psock_backlog(struct work_struct *work) * other work that might be here. */ if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) - schedule_delayed_work(&psock->work, 1); + sk_psock_schedule_delayed_work(psock, 1); goto end; } /* Hard errors break pipe and stop xmit. */ @@ -940,6 +940,7 @@ static int sk_psock_skb_redirect(struct sk_psock *from, struct sk_buff *skb) sock_drop(from->sk, skb); return -EIO; } + psock_other->redir_cpu = from->redir_cpu; spin_lock_bh(&psock_other->ingress_lock); if (!sk_psock_test_state(psock_other, SK_PSOCK_TX_ENABLED)) { spin_unlock_bh(&psock_other->ingress_lock); @@ -949,7 +950,7 @@ static int sk_psock_skb_redirect(struct sk_psock *from, struct sk_buff *skb) }
skb_queue_tail(&psock_other->ingress_skb, skb); - schedule_delayed_work(&psock_other->work, 0); + sk_psock_schedule_delayed_work(psock_other, 0); spin_unlock_bh(&psock_other->ingress_lock); return 0; } @@ -1027,7 +1028,7 @@ static int sk_psock_verdict_apply(struct sk_psock *psock, struct sk_buff *skb, spin_lock_bh(&psock->ingress_lock); if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) { skb_queue_tail(&psock->ingress_skb, skb); - schedule_delayed_work(&psock->work, 0); + sk_psock_schedule_delayed_work(psock, 0); err = 0; } spin_unlock_bh(&psock->ingress_lock); @@ -1059,7 +1060,7 @@ static void sk_psock_write_space(struct sock *sk) psock = sk_psock(sk); if (likely(psock)) { if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) - schedule_delayed_work(&psock->work, 0); + sk_psock_schedule_delayed_work(psock, 0); write_space = psock->saved_write_space; } rcu_read_unlock();
Add cpu-affinity for sockmap bench. Also add no-verify args to avoid validating data for performance enhancements.
Signed-off-by: Jiayuan Chen jiayuan.chen@linux.dev --- .../selftests/bpf/benchs/bench_sockmap.c | 35 +++++++++++++++++-- tools/testing/selftests/bpf/bpf_kfuncs.h | 6 ++++ .../selftests/bpf/progs/bench_sockmap_prog.c | 7 ++++ 3 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/benchs/bench_sockmap.c b/tools/testing/selftests/bpf/benchs/bench_sockmap.c index 8ebf563a67a2..e004a618822a 100644 --- a/tools/testing/selftests/bpf/benchs/bench_sockmap.c +++ b/tools/testing/selftests/bpf/benchs/bench_sockmap.c @@ -43,6 +43,8 @@ enum SOCKMAP_ARG_FLAG { ARG_FW_TX_VERDICT_INGRESS, ARG_FW_TX_VERDICT_EGRESS, ARG_CTL_RX_STRP, + ARG_CTL_CPU_AFFINITY, + ARG_CTL_NO_VERIFY, ARG_CONSUMER_DELAY_TIME, ARG_PRODUCER_DURATION, }; @@ -109,6 +111,8 @@ static struct socmap_ctx { int delay_consumer; int prod_run_time; int strp_size; + int cpu_affinity; + int skip_verify; } ctx = { .prod_send = 0, .user_read = 0, @@ -118,6 +122,8 @@ static struct socmap_ctx { .delay_consumer = 0, .prod_run_time = 0, .strp_size = 0, + .cpu_affinity = 0, + .skip_verify = 0, };
static void bench_sockmap_prog_destroy(void) @@ -235,11 +241,18 @@ static int create_sockets(void) static void validate(void) { if (env.consumer_cnt != 2 || env.producer_cnt != 1 || - !env.affinity) + !env.affinity) { + fprintf(stderr, "argument '-c 2 -p 1 -a' is necessary\n"); goto err; + } + + if (!ctx.cpu_affinity && env.nr_cpus < 4) { + fprintf(stderr, "4 CPU are needed to test cpu-affinity\n"); + goto err; + } + return; err: - fprintf(stderr, "argument '-c 2 -p 1 -a' is necessary"); exit(1); }
@@ -327,6 +340,9 @@ static void setup(void) exit(1); }
+ if (ctx.cpu_affinity) + ctx.skel->data->redir_cpu = 3; + if (create_sockets()) { fprintf(stderr, "create_net_mode error\n"); goto err; @@ -367,9 +383,12 @@ static void measure(struct bench_res *res)
static void verify_data(int *check_pos, char *buf, int rcv) { + if (ctx.skip_verify) + return; + for (int i = 0 ; i < rcv; i++) { if (buf[i] != snd_data[(*check_pos) % DATA_REPEAT_SIZE]) { - fprintf(stderr, "verify data fail"); + fprintf(stderr, "verify data fail\n"); exit(1); } (*check_pos)++; @@ -553,6 +572,10 @@ static const struct argp_option opts[] = { "delay consumer start"}, { "producer-duration", ARG_PRODUCER_DURATION, "SEC", 0, "producer duration"}, + { "cpu-affinity", ARG_CTL_CPU_AFFINITY, NULL, 0, + "set cpu-affinity for sockmap backlog thread"}, + { "no-verify", ARG_CTL_NO_VERIFY, NULL, 0, + "skip data validation for performance enhancements"}, {}, };
@@ -571,6 +594,12 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) case ARG_CTL_RX_STRP: ctx.strp_size = strtol(arg, NULL, 10); break; + case ARG_CTL_CPU_AFFINITY: + ctx.cpu_affinity = 1; + break; + case ARG_CTL_NO_VERIFY: + ctx.skip_verify = 1; + break; default: return ARGP_ERR_UNKNOWN; } diff --git a/tools/testing/selftests/bpf/bpf_kfuncs.h b/tools/testing/selftests/bpf/bpf_kfuncs.h index 8215c9b3115e..173329c5d034 100644 --- a/tools/testing/selftests/bpf/bpf_kfuncs.h +++ b/tools/testing/selftests/bpf/bpf_kfuncs.h @@ -92,4 +92,10 @@ extern int bpf_set_dentry_xattr(struct dentry *dentry, const char *name__str, const struct bpf_dynptr *value_p, int flags) __ksym __weak; extern int bpf_remove_dentry_xattr(struct dentry *dentry, const char *name__str) __ksym __weak;
+/* Description + * Set sockmap redir cpu + * Returns + * Error code + */ +extern int bpf_sk_skb_set_redirect_cpu(struct __sk_buff *skb, int redir_cpu) __ksym; #endif diff --git a/tools/testing/selftests/bpf/progs/bench_sockmap_prog.c b/tools/testing/selftests/bpf/progs/bench_sockmap_prog.c index 079bf3794b3a..dd1a11cb4f48 100644 --- a/tools/testing/selftests/bpf/progs/bench_sockmap_prog.c +++ b/tools/testing/selftests/bpf/progs/bench_sockmap_prog.c @@ -2,11 +2,15 @@ #include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> +#include <stdbool.h> +#include "bpf_kfuncs.h"
long process_byte = 0; int verdict_dir = 0; int dropped = 0; int pkt_size = 0; +int redir_cpu = -1; + struct { __uint(type, BPF_MAP_TYPE_SOCKMAP); __uint(max_entries, 20); @@ -33,6 +37,9 @@ int prog_skb_verdict(struct __sk_buff *skb) int one = 1; int ret = bpf_sk_redirect_map(skb, &sock_map_rx, one, verdict_dir);
+ if (redir_cpu != -1) + bpf_sk_skb_set_redirect_cpu(skb, redir_cpu); + if (ret == SK_DROP) dropped++; __sync_fetch_and_add(&process_byte, skb->len);
On Mon, Apr 28, 2025 at 1:18 AM Jiayuan Chen jiayuan.chen@linux.dev wrote:
Abstract
This patchset improves the performance of sockmap by providing CPU affinity, resulting in a 1-10x increase in throughput.
Motivation
Traditional user-space reverse proxy:
Reserve Proxy _________________client -> | fd1 <-> fd2 | -> server |_________________|
Using sockmap for reverse proxy:
Reserve Proxy _________________client -> | fd1 <-> fd2 | -> server | |_________________| | | | | | | _________ | | | sockmap | | --> |_________| -->
By adding fds to sockmap and using a BPF program, we can quickly forward data and avoid data copying between user space and kernel space.
Mainstream multi-process reverse proxy applications, such as Nginx and HAProxy, support CPU affinity settings, which allow each process to be pinned to a specific CPU, avoiding conflicts between data plane processes and other processes, especially in multi-tenant environments.
Current Issues
The current design of sockmap uses a workqueue to forward ingress_skb and wakes up the workqueue without specifying a CPU (by calling schedule_delayed_work()). In the current implementation of schedule_delayed_work, it tends to run the workqueue on the current CPU.
This approach has a high probability of running on the current CPU, which is the same CPU that handles the net rx soft interrupt, especially for programs that access each other using local interfaces.
The loopback driver's transmit interface, loopback_xmit(), directly calls __netif_rx() on the current CPU, which means that the CPU handling sockmap's workqueue and the client's sending CPU are the same, resulting in contention.
For a TCP flow, if the request or response is very large, the psock->ingress_skb queue can become very long. When the workqueue traverses this queue to forward the data, it can consume a significant amount of CPU time.
Solution
Configuring RPS on a loopback interface can be useful, but it will trigger additional softirq, and furthermore, it fails to achieve our expected effect of CPU isolation from other processes.
Instead, we provide a kfunc that allow users to specify the CPU on which the workqueue runs through a BPF program.
We can use the existing benchmark to test the performance, which allows us to evaluate the effectiveness of this optimization.
Because we use local interfaces for communication and the client consumes a significant amount of CPU when sending data, this prevents the workqueue from processing ingress_skb in a timely manner, ultimately causing the server to fail to read data quickly.
Without cpu-affinity: ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 36.031us): Send Speed 1143.693 MB/s ... Rcv Speed 109.572 MB/s Iter 1 ( 0.608us): Send Speed 1320.550 MB/s ... Rcv Speed 48.103 MB/s Iter 2 ( -5.448us): Send Speed 1314.790 MB/s ... Rcv Speed 47.842 MB/s Iter 3 ( -0.613us): Send Speed 1320.158 MB/s ... Rcv Speed 46.531 MB/s Iter 4 ( -3.441us): Send Speed 1319.375 MB/s ... Rcv Speed 46.662 MB/s Iter 5 ( 3.764us): Send Speed 1166.667 MB/s ... Rcv Speed 42.467 MB/s Iter 6 ( -4.404us): Send Speed 1319.508 MB/s ... Rcv Speed 47.973 MB/s Summary: total trans 7758 MB ± 1293.506 MB/s
Without cpu-affinity(RPS enabled): ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 28.925us): Send Speed 1630.357 MB/s ... Rcv Speed 850.960 MB/s Iter 1 ( -2.042us): Send Speed 1644.564 MB/s ... Rcv Speed 822.478 MB/s Iter 2 ( 0.754us): Send Speed 1644.297 MB/s ... Rcv Speed 850.787 MB/s Iter 3 ( 0.159us): Send Speed 1644.429 MB/s ... Rcv Speed 850.198 MB/s Iter 4 ( -2.898us): Send Speed 1646.924 MB/s ... Rcv Speed 830.867 MB/s Iter 5 ( -0.210us): Send Speed 1649.410 MB/s ... Rcv Speed 824.246 MB/s Iter 6 ( -1.448us): Send Speed 1650.723 MB/s ... Rcv Speed 808.256 MB/s
With cpu-affinity(RPS disabled): ./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify --cpu-affinity Setting up benchmark 'sockmap'... create socket fd c1:14 p1:15 c2:16 p2:17 Benchmark 'sockmap' started. Iter 0 ( 36.051us): Send Speed 1883.437 MB/s ... Rcv Speed 1865.087 MB/s Iter 1 ( 1.246us): Send Speed 1900.542 MB/s ... Rcv Speed 1761.737 MB/s Iter 2 ( -8.595us): Send Speed 1883.128 MB/s ... Rcv Speed 1860.714 MB/s Iter 3 ( 7.033us): Send Speed 1890.831 MB/s ... Rcv Speed 1806.684 MB/s Iter 4 ( -8.397us): Send Speed 1884.700 MB/s ... Rcv Speed 1973.568 MB/s Iter 5 ( -1.822us): Send Speed 1894.125 MB/s ... Rcv Speed 1775.046 MB/s Iter 6 ( 4.936us): Send Speed 1877.597 MB/s ... Rcv Speed 1959.320 MB/s Summary: total trans 11328 MB ± 1888.507 MB/s
This looks to me like an artificial benchmark. Surely perf will be higher when wq is executed on free cpu. In production all cpus likely have work to do, so this whole approach 'lets ask wq to run on that cpu' isn't going to work. Looks like RPS helps. Use that. I think it will scale and work better when the whole server is loaded.
pw-bot: cr
April 30, 2025 at 07:26, "Alexei Starovoitov" alexei.starovoitov@gmail.com wrote:
On Mon, Apr 28, 2025 at 1:18 AM Jiayuan Chen jiayuan.chen@linux.dev wrote:
Abstract
===
This patchset improves the performance of sockmap by providing CPU affinity,
resulting in a 1-10x increase in throughput.
Motivation
===
Traditional user-space reverse proxy:
Reserve Proxy
client -> | fd1 <-> fd2 | -> server
|_________________|
Using sockmap for reverse proxy:
Reserve Proxy
client -> | fd1 <-> fd2 | -> server
| |_________________| |
| | | |
| _________ |
| | sockmap | |
--> |_________| -->
By adding fds to sockmap and using a BPF program, we can quickly forward
data and avoid data copying between user space and kernel space.
Mainstream multi-process reverse proxy applications, such as Nginx and
HAProxy, support CPU affinity settings, which allow each process to be
pinned to a specific CPU, avoiding conflicts between data plane processes
and other processes, especially in multi-tenant environments.
Current Issues
===
The current design of sockmap uses a workqueue to forward ingress_skb and
wakes up the workqueue without specifying a CPU
(by calling schedule_delayed_work()). In the current implementation of
schedule_delayed_work, it tends to run the workqueue on the current CPU.
This approach has a high probability of running on the current CPU, which
is the same CPU that handles the net rx soft interrupt, especially for
programs that access each other using local interfaces.
The loopback driver's transmit interface, loopback_xmit(), directly calls
__netif_rx() on the current CPU, which means that the CPU handling
sockmap's workqueue and the client's sending CPU are the same, resulting
in contention.
For a TCP flow, if the request or response is very large, the
psock->ingress_skb queue can become very long. When the workqueue
traverses this queue to forward the data, it can consume a significant
amount of CPU time.
Solution
===
Configuring RPS on a loopback interface can be useful, but it will trigger
additional softirq, and furthermore, it fails to achieve our expected
effect of CPU isolation from other processes.
Instead, we provide a kfunc that allow users to specify the CPU on which
the workqueue runs through a BPF program.
We can use the existing benchmark to test the performance, which allows
us to evaluate the effectiveness of this optimization.
Because we use local interfaces for communication and the client consumes
a significant amount of CPU when sending data, this prevents the workqueue
from processing ingress_skb in a timely manner, ultimately causing the
server to fail to read data quickly.
Without cpu-affinity:
./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify
Setting up benchmark 'sockmap'...
create socket fd c1:14 p1:15 c2:16 p2:17
Benchmark 'sockmap' started.
Iter 0 ( 36.031us): Send Speed 1143.693 MB/s ... Rcv Speed 109.572 MB/s
Iter 1 ( 0.608us): Send Speed 1320.550 MB/s ... Rcv Speed 48.103 MB/s
Iter 2 ( -5.448us): Send Speed 1314.790 MB/s ... Rcv Speed 47.842 MB/s
Iter 3 ( -0.613us): Send Speed 1320.158 MB/s ... Rcv Speed 46.531 MB/s
Iter 4 ( -3.441us): Send Speed 1319.375 MB/s ... Rcv Speed 46.662 MB/s
Iter 5 ( 3.764us): Send Speed 1166.667 MB/s ... Rcv Speed 42.467 MB/s
Iter 6 ( -4.404us): Send Speed 1319.508 MB/s ... Rcv Speed 47.973 MB/s
Summary: total trans 7758 MB ± 1293.506 MB/s
Without cpu-affinity(RPS enabled):
./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify
Setting up benchmark 'sockmap'...
create socket fd c1:14 p1:15 c2:16 p2:17
Benchmark 'sockmap' started.
Iter 0 ( 28.925us): Send Speed 1630.357 MB/s ... Rcv Speed 850.960 MB/s
Iter 1 ( -2.042us): Send Speed 1644.564 MB/s ... Rcv Speed 822.478 MB/s
Iter 2 ( 0.754us): Send Speed 1644.297 MB/s ... Rcv Speed 850.787 MB/s
Iter 3 ( 0.159us): Send Speed 1644.429 MB/s ... Rcv Speed 850.198 MB/s
Iter 4 ( -2.898us): Send Speed 1646.924 MB/s ... Rcv Speed 830.867 MB/s
Iter 5 ( -0.210us): Send Speed 1649.410 MB/s ... Rcv Speed 824.246 MB/s
Iter 6 ( -1.448us): Send Speed 1650.723 MB/s ... Rcv Speed 808.256 MB/s
With cpu-affinity(RPS disabled):
./bench sockmap -c 2 -p 1 -a --rx-verdict-ingress --no-verify --cpu-affinity
Setting up benchmark 'sockmap'...
create socket fd c1:14 p1:15 c2:16 p2:17
Benchmark 'sockmap' started.
Iter 0 ( 36.051us): Send Speed 1883.437 MB/s ... Rcv Speed 1865.087 MB/s
Iter 1 ( 1.246us): Send Speed 1900.542 MB/s ... Rcv Speed 1761.737 MB/s
Iter 2 ( -8.595us): Send Speed 1883.128 MB/s ... Rcv Speed 1860.714 MB/s
Iter 3 ( 7.033us): Send Speed 1890.831 MB/s ... Rcv Speed 1806.684 MB/s
Iter 4 ( -8.397us): Send Speed 1884.700 MB/s ... Rcv Speed 1973.568 MB/s
Iter 5 ( -1.822us): Send Speed 1894.125 MB/s ... Rcv Speed 1775.046 MB/s
Iter 6 ( 4.936us): Send Speed 1877.597 MB/s ... Rcv Speed 1959.320 MB/s
Summary: total trans 11328 MB ± 1888.507 MB/s
This looks to me like an artificial benchmark. Surely perf will be higher when wq is executed on free cpu. In production all cpus likely have work to do, so this whole approach 'lets ask wq to run on that cpu' isn't going to work. Looks like RPS helps. Use that. I think it will scale and work better when the whole server is loaded. pw-bot: cr
Hi Alexei, you're right for requests coming from a remote host, all CPUs are running; in cloud-native scenarios where Sidecars are widely used, they access each other through loopback, but for requests accessing each other through loopback, the wq (workqueue) will definitely run on the CPU where the client is located (based on the implementation of loopback and wq). Since the Sidecar itself is bound to a CPU, which means that in actual scenarios, the CPU bound to the gateway (reverse proxy) program using sockmap cannot be fully utilized.
Enabling RPS can alleviate the sockmap issue, but it will introduce an extra software calculation, so from a performance perspective, we still expect to have a solution that can achieve the highest performance.
Thanks.
On Tue, Apr 29, 2025 at 4:47 PM Jiayuan Chen jiayuan.chen@linux.dev wrote:
This looks to me like an artificial benchmark. Surely perf will be higher when wq is executed on free cpu. In production all cpus likely have work to do, so this whole approach 'lets ask wq to run on that cpu' isn't going to work. Looks like RPS helps. Use that. I think it will scale and work better when the whole server is loaded. pw-bot: cr
Hi Alexei, you're right for requests coming from a remote host, all CPUs are running; in cloud-native scenarios where Sidecars are widely used, they access each other through loopback, but for requests accessing each other through loopback, the wq (workqueue) will definitely run on the CPU where the client is located (based on the implementation of loopback and wq). Since the Sidecar itself is bound to a CPU, which means that in actual scenarios, the CPU bound to the gateway (reverse proxy) program using sockmap cannot be fully utilized.
Enabling RPS can alleviate the sockmap issue, but it will introduce an extra software calculation, so from a performance perspective, we still expect to have a solution that can achieve the highest performance.
And I think it's wrong to optimize for performance of one particular setup. An API that picks a cpu is difficult to get right. Too easy to make performance worse.
linux-kselftest-mirror@lists.linaro.org
-
Alexei Starovoitov -
Cong Wang -
Jiayuan Chen