On Sun, Mar 28, 2021 at 6:40 PM Alexei Starovoitov alexei.starovoitov@gmail.com wrote:
On Sat, Mar 27, 2021 at 09:32:58PM -0700, Andrii Nakryiko wrote:
I think it's better to start with new library for tc/xdp and have libbpf as a dependency on that new lib. For example we can add it as subdir in tools/lib/bpf/.
Similarly I think integerating static linking into libbpf was a mistake. It should be a sub library as well.
If we end up with core libbpf and ten sublibs for tc, xdp, af_xdp, linking, whatever else the users would appreciate that we don't shove single libbpf to them with a ton of features that they might never use.
What's the concern exactly? The size of the library? Having 10 micro-libraries has its own set of downsides,
specifically?
You didn't answer my question, but from what you write below I assume libbpf size is your main concern?
As for downsides, I'm sure I'm not yet seeing all of the problems we'll encounter when splitting libbpf into 10 pieces. But as a user, having to figure out which libraries I need to use is a big hassle. E.g., for XDP application using ringbuf, I'll need libbpfelf, libbpftrace, libbpfnet, which implicitly also would depend on libsysbpf, libbtf, libbpfutil, I assume. So having to list 3 vs 1 library is already annoying, but when statically linking I'd need to specify all 6. I'd very much rather know that it has to be -lbpf at it will provide me with all the basics (and it's already -lz and -lelf in static linking scenario, which I wish we could get rid of).
I'm not convinced that's a better situation for end users. And would certainly cause more hassle for libbpf developers and packagers.
For developers and packagers.. yes. For users.. quite the opposite.
See above. I don't know which hassle is libbpf for users today. You were implying code size used for functionality users might not use (e.g., linker). Libbpf is a very small library, <300KB. There are users building tools for constrained embedded systems that use libbpf. There are/were various problems mentioned, but the size of libbpf wasn't yet one of them. We should certainly watch the code bloat, but we are not yet at the point where library is too big for users to be turned off. In shared library case it's even less of a concern.
The skel gen and static linking must be split out before the next libbpf release. Not a single application linked with libbpf is going to use those pieces. bpftool is one and only that needs them. Hence forcing libbpf users to increase their .text with a dead code is a selfish call of libbpf developers and packagers. The user's priorities must come first.
And what did you include in "core libbpf"?
I would take this opportunity to split libbpf into maintainable pieces:
- libsysbpf - sys_bpf wrappers (pretty much tools/lib/bpf/bpf.c)
- libbpfutil - hash, strset
strset and hash are internal data structures, I never intended to expose them through public APIs. I haven't investigated, but if we have a separate shared library (libbpfutil), I imagine we won't be able to hide those APIs, right?
- libbtf - BTF read/write
- libbpfelf - ELF parsing, CORE, ksym, kconfig
- libbpfskel - skeleton gen used by bpftool only
skeleton generation is already part of bpftool, there is no need to split anything out
- libbpflink - linker used by bpftool only
- libbpfnet - networking attachment via netlink including TC and XDP
- libbpftrace - perfbuf, ringbuf
ringbuf and perfbuf are both very small code-wise, and are used in majority of BPF applications anyways
- libxdp - Toke's xdp chaining
- libxsk - af_xdp logic
Now, if we look at libbpf .o files, we can approximately see what functionality is using most code:
File Size Percent
bpf.o 17800 4.88 bpf_prog_linfo.o 2952 0.81 btf_dump.o 20472 5.61 btf.o 58160 15.93 hashmap.o 4056 1.11 libbpf_errno.o 2912 0.80 libbpf.o 190072 52.06 libbpf_probes.o 6696 1.83 linker.o 29408 8.05 netlink.o 5944 1.63 nlattr.o 2744 0.75 ringbuf.o 6128 1.68 str_error.o 1640 0.45 strset.o 3656 1.00 xsk.o 12456 3.41
Total 365096 100.00
so libbpf.o which has mostly bpf_object open/load logic and CO-RE take more than half already. And it depends on still more stuff in btf, hashmap, bpf, libbpf_probes, errno. But the final code size is even smaller, because libbpf.so is just 285128 bytes (not 365096 as implied by the table above), so even these numbers are pessimistic.
linker.o, which is about 8% of the code right now, but is also actually taking less than 29KB, because when I remove linker.o and re-compile, the final libbpf.so goes from 285128 to 267576 = 17552 reduction. Even if it grows 2x, I'd still say it's not a big deal.
One reason to keep BPF linker in libbpf is that it is not only bpftool that would be using it. Our libbpf Rust bindings is implementing its own BPF skeleton generation, and we'd like to use linker APIs to support static linking when using libbpf-rs without depending on bpftool. So having it in libbpf and not in bpftool is good when you consider the wider ecosystem.
But again, let's just reflect for a second that we are talking about the library that takes less than 300KB total. It would be also interesting to experiment with LTO and its effect on final binaries when statically linking against libbpf. I haven't tried yet, though.
In the future the stack trace symbolization code can come into libbpftrace or be a part of its own lib. My upcoming loader program and signed prog generation logic can be part of libbpfskel.