On Wed, 11 Nov 2026 10:48:47 +0000 Matt Evans matt@ozlabs.org wrote:
Hi Alex,
On 14/10/2026 21:38, Matt Evans wrote:
Hi Alexes,
On 14/10/2026 00:13, Alex Williamson wrote:
On Tue, 6 Oct 2026 20:36:36 +0100 Matt Evans matt@ozlabs.org wrote:
A VFIO DMABUF can export a subset of a BAR to userspace by fd; add support for mmap() of this fd. This provides another route for a process to map BARs, in which the process can only map a specific subset of a BAR represented by the exported DMABUF.
mmap() support enables userspace drivers to safely delegate BAR sub-ranges to other processes by sharing a DMABUF fd, without having to share the (omnipotent) VFIO device fd with them.
Since the main VFIO BAR mmap() is now DMABUF-aware, the new mmap() reuses the existing vm_ops.
The lifecycle of an exported DMABUF remains decoupled from that of the device fd it came from, i.e. the device fd could be closed with DMABUF VMAs present, meaning a fault on a VMA could happen concurrently with vfio_pci_dma_buf_cleanup().
To deal with this scenario, the fault handler now temporarily takes a VFIO device registration to ensure the vdev remains valid, and then vdev locks can be taken.
This sounds like potentially measurable overhead if something is slamming the fault handler where huge-fault is not supported or active. Have you considered a fast path for the internal dmabuf where the lifecycle is constrained?
[snip]
Hmm. I'll hack up a test to try to measure the relative overhead of VMA faults before/after (modulo access to meaningful HW).
Alex Mastro and I have put together some tests, and measured this overhead on the v8 series.
https://github.com/evansm7/vfio-playground/tree/master/vfio-barmap-perf
Alex also generated a similar test with a slightly different method: https://github.com/opsound/vfio-tools/blob/main/vfio_bar_bench.c
Test method:
Time accesses to a VFIO mmap'd BAR immediately after the mmap, to measure the 'cold' initial fault time. Re-time a second pass for which the pagetable should now be populated, and subtract. This should isolate the cost of the fault (uarch exception, generic pagefault plumbing, VFIO fault handler) from common costs (the MMIO access itself, TLB misses, the loop iterating across the BAR). Maximise the number of faults by disabling hugepages (per concern above).
This is done both for a single-threaded case which just iterates across a new BAR mapping, and a multi-threaded contended case. Rather than just pthreads, this test fork()s to try to avoid seeing contention on per-mm locks (e.g. mmap_lock) and to focus on the VMA/VFIO locking.
The default test mmaps before forking, meaning there's one shared mapping/DMABUF accessed from both processes. The fault handler should then contend on dmabuf->resv, vdev->memory_lock, and vdev->dmabuf_lock.
For comparison/curiosity, the --mt-map-late option mmaps _after_ forking; this leads to a mapping/DMABUF in each process so I expect no contention on dmabuf->resv (but still on memory_lock/dmabuf_lock).
Alex's test instead runs threads in-process, and goes further with supporting a larger number of threads (though with the caveat that they may also be fighting over intra-process mm locks). It also supports a mode where threads can access per-thread VMAs (which is comparable to the 'mt-map-late' option of my test). Both are interesting approaches IMO.
Findings:
On the dusty AMD Ryzen 3950X under my desk (targeting a 256MB BAR) I found the following averaged over 200 runs:
- A single-threaded/uncontended fault became 5.8% slower
- A two-process contended case became 8.7% slower
- 0.5% of this is the "early mmap" dmabuf->resv contention, i.e. using mt-map-late (a mapping in each process) reduces by 0.5%.
Alex's runs on fancier EPYC 9334 machines show a similar (slightly lower) overhead in the 1-2 case, and go on to show fairly high overhead when many threads are involved (baseline times in ns, relative series v8 behaviour in percentage change):
Threads One shared mapping Mapping per thread 7.3 v8 7.3 v8
1 761 +4.5% 777 +2.7% 2 869 +6.6% 840 +5.6% 8 1465 +78% 1363 +47% 16 4152 +97% 4258 +19% 32 9927 +92% 10158 +12%Now. The absolute timings of these sub-microsecond faults indicate a realistic application won't see a human-visible change. Small BARs mapped with 4k pages are still fast to map; it is reasonable to ask that applications map huge (GBs/TBs) BARs using huge (MB/GB) pages! Of course, this is just a one-time cost of warming up the VMA, not ongoing.
Alex, my preference is not to complicate the fault handler with a fast path just yet. It's easy to think of a contrived scenario where this matters but in middle-of-the-road application behaviour it would be surprising if this was noticeable. What are your thoughts?
(In the previous mail I suggested keying off priv->vfile (or a flag thereof), and it seems ~straightforward to skip the extra synchronisation. If you feel strongly that the fault handler doesn't have enough if() statements yet and needs more paths then we could bring this into v9, but would prefer to follow up later. :) A fastpath diff appears small but the reasoning overhead is bigger.)
I tend to agree. IIUC, in the worst case concurrency (that we likely don't experience), if we assume huge pages, we're looking at low single digit millisecond total overhead to fault in an entire BAR (~256 max faults). Further, if our configuration doesn't force type1 to fault in the entire BAR for DMA mapping, that overhead is amortized across every user/guest access.
So yes, it's probably not worth having two different locking semantics, but we should note that as an intentional decision in the commit log, potentially even as a comment in the fault handler. Thanks,
Alex