On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
Jason
Jason Gunthorpe jgg@nvidia.com writes:
On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
I'd like to propose an alternative. I've been working on guest_memfd and new to the world of IO, please help me along! :)
It seems like using dmabufs are used a little awkwardly here. IIUC dmabufs were originally meant to expose memory of one device to another device, mostly meant to share memory. Dmabufs do expose MMIO too, for device to device communications. Without virtualization, userspace MMIO would be done by mmap()-ing a VFIO fd and having the userspace program write to the userspace addresses.
Before CoCo, device passthrough (MMIO) is mostly handled by mmap()-ing a VFIO fd and setting up the userspace address in a KVM memslot for the guest.
With CoCo, is the problem we're solving that we want KVM to know what pfns to set up in stage 2 page tables, but not via userspace addresses?
guest_memfd already does that for regular host memory, tracks the private/shared-ness of the memory, tracks which struct kvm the memory belongs to.
guest_memfd functions as KVM's bridge to host memory. KVM already can ask guest_memfd for the pfn to map into stage 2 page tables, and already asks guest_memfd for the shared/private state of the memory. guest_memfd already also blocks the host from faulting guest private memory (mmap()-ing is always allowed).
Instead of using dmabuf as the intermediary between the MMIO PFNs and KVM, why not use guest_memfd?
What if we make guest_memfd accept a VFIO fd, or a dmabuf fd?
guest_memfd can then take the mmap() calls from userspace and .fault() from mm, and then forward them to VFIO or dmabuf. This way, VFIO/dmabuf can stick to their original functions, and the changes to VFIO/dmabuf would probably revolve around disabling access.
Disabling access would probably involve some of these:
+ When guest_memfd receives the fd, it could return error for existing mappings, or perhaps it could just force-unmap. + 1 extra flag or field to indicate that guest_memfd is controlling this file, so that if userspace tries to take some actions with the original VFIO or dmabuf fd, the request should be blocked. + Perhaps just close the original fd, like dup2(oldfd, newfd) closes newfd?
I'm about to restart work on guest_memfd HugeTLB and I'm thinking about a similar approach for guest_memfd HugeTLB, where perhaps the interface could be that userspace will give guest_memfd a HugeTLB fd at creation time, and then the original HugeTLB fd would be rendered unusable in the same way as above, perhaps like with the S_IMMUTABLE inode flag, but also blocking reads, and not userspace-modifiable.
In the course of a CoCo guest's operation, will the guest need to convert between private/shared MMIO? Will the guest need some pages shared and others private? If these are required operations, guest_memfd already provides the tracking and is going to have a conversion ioctl very soon. Instead of further extending dmabuf to track more things, how about letting guest_memfd track it?
On Sat, Jul 11, 2026 at 06:01:31PM -0700, Ackerley Tng wrote:
In the course of a CoCo guest's operation, will the guest need to convert between private/shared MMIO? Will the guest need some pages shared and others private? If these are required operations, guest_memfd already provides the tracking and is going to have a conversion ioctl very soon. Instead of further extending dmabuf to track more things, how about letting guest_memfd track it?
Use another FD type was sort of my fallback if we couldn't get DMABUF into something workable. I'm kind of surprised to see guestmemfd proposed as the other FD, but I don't know much about its insides.
If VFIO can create one and fill it with MMIO physical addresses then maybe it is OK?
Jason
Jason Gunthorpe jgg@nvidia.com writes:
On Sat, Jul 11, 2026 at 06:01:31PM -0700, Ackerley Tng wrote:
In the course of a CoCo guest's operation, will the guest need to convert between private/shared MMIO? Will the guest need some pages shared and others private? If these are required operations, guest_memfd already provides the tracking and is going to have a conversion ioctl very soon. Instead of further extending dmabuf to track more things, how about letting guest_memfd track it?
Use another FD type was sort of my fallback if we couldn't get DMABUF into something workable. I'm kind of surprised to see guestmemfd proposed as the other FD, but I don't know much about its insides.
For now guest_memfd only supports 4K pages, so the interfaces we're interested in are:
+ The .fault() handler, which core mm calls to service any page faults. For this, guest_memfd first checks if the page is shared, and if so, proceeds to return a struct page. + For VFIO, guest_memfd could call vfio_pci_mmap_huge_fault() and do vmf_insert_pfn(). + The kvm_gmem_get_pfn() function, which kvm calls to get a pfn and a max_order to set up stage 2 page tables. Now guest_memfd returns folio_pfn(folio) and max_order = 0 since guest_memfd only supports PAGE_SIZE pages now. + For VFIO, guest_memfd would need some way to get the PFN, so probably something like dma_buf_get_pfn_unlocked() [1], but probably from VFIO instead of dmabuf. Perhaps pci_resource_start() is good enough to get a pfn.
[1] https://lore.kernel.org/all/20250529053513.1592088-2-yilun.xu@linux.intel.co...
If VFIO can create one
At guest_memfd creation time, guest_memfd is always created for a KVM. This is the first place where it is bound to a kvm instance. This is one place where it helps Yilun with Confidential VMs, so KVM can be sure that the memory was meant for a specific CoCo kvm and not any VM.
I think it might be weird to have VFIO take a kvm fd to create a guest_memfd?
I was thinking that the userspace VMM could do something like:
int vfio_fd = open("/dev/vfio/devices/vfio0"); int gmem_fd = ioctl(KVM_CREATE_GUEST_MEMFD, { .fd = vfio_fd });
and creating from a template fd would also disable/close the vfio_fd so that the following will fail:
addr = mmap(vfio_fd); // Maybe this should fail too? *addr = 1; // But this should definitely fail so that // guest_memfd gets to block private writes.
and fill it with MMIO physical addresses then maybe it is OK?
Jason
What interface would be used fill a guest_memfd with MMIO physical addresses?
When a vdev is associated with a vfio file, can some other vfio file be created that can also do MMIO to the vdev's PFNs? Would I be able to set something to block any future associations to those PFNs? This will help guest_memfd to prevent any other writes to private PFNs.
On Sun, 12 Jul 2026 at 02:02, Ackerley Tng ackerleytng@google.com wrote:
Jason Gunthorpe jgg@nvidia.com writes:
On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
I'd like to propose an alternative. I've been working on guest_memfd and new to the world of IO, please help me along! :)
It seems like using dmabufs are used a little awkwardly here. IIUC dmabufs were originally meant to expose memory of one device to another device, mostly meant to share memory. Dmabufs do expose MMIO too, for device to device communications. Without virtualization, userspace MMIO would be done by mmap()-ing a VFIO fd and having the userspace program write to the userspace addresses.
Before CoCo, device passthrough (MMIO) is mostly handled by mmap()-ing a VFIO fd and setting up the userspace address in a KVM memslot for the guest.
With CoCo, is the problem we're solving that we want KVM to know what pfns to set up in stage 2 page tables, but not via userspace addresses?
guest_memfd already does that for regular host memory, tracks the private/shared-ness of the memory, tracks which struct kvm the memory belongs to.
guest_memfd functions as KVM's bridge to host memory. KVM already can ask guest_memfd for the pfn to map into stage 2 page tables, and already asks guest_memfd for the shared/private state of the memory. guest_memfd already also blocks the host from faulting guest private memory (mmap()-ing is always allowed).
Instead of using dmabuf as the intermediary between the MMIO PFNs and KVM, why not use guest_memfd?
What if we make guest_memfd accept a VFIO fd, or a dmabuf fd?
This is interesting for pKVM too, provided it covers more than MMIO.
We need guest_memfd to be backable by a dmabuf for ordinary guest memory, not only for device MMIO. There is mobile hardware that doesn't tolerate scattered private memory (DMA engines that can't gather, IOMMU page-table size constraints), and a CMA-backed dmabuf heap is the practical way to get contiguous memory at runtime. HugeTLB doesn't help, it wants boot-time reservation. Those pages are struct-page backed, so it's a different problem from the non-struct-page MMIO case, and the shared parts still need to be GUP-able.
More important for the API shape: conversions have to work on subsets of such a region, at page granularity. A pKVM guest doesn't know what backs its memory, so it will issue share/unshare hypercalls over arbitrary ranges of whatever it was given. If a dmabuf-backed guest_memfd can only be converted as a whole, we can't use it for memory, and the guest can't be taught to care.
Quentin made both points on the in-place conversion series [1], and covered the wider framing at KVM Forum [2].
Cheers, /fuad
[1] https://lore.kernel.org/all/od4dx6snqsl2qiocgf3jxm4dndxhrlvsfr22eveuno6nskgf... [2] https://www.youtube.com/watch?v=zaBxoyRepzA
guest_memfd can then take the mmap() calls from userspace and .fault() from mm, and then forward them to VFIO or dmabuf. This way, VFIO/dmabuf can stick to their original functions, and the changes to VFIO/dmabuf would probably revolve around disabling access.
Disabling access would probably involve some of these:
- When guest_memfd receives the fd, it could return error for existing mappings, or perhaps it could just force-unmap.
- 1 extra flag or field to indicate that guest_memfd is controlling this file, so that if userspace tries to take some actions with the original VFIO or dmabuf fd, the request should be blocked.
- Perhaps just close the original fd, like dup2(oldfd, newfd) closes newfd?
I'm about to restart work on guest_memfd HugeTLB and I'm thinking about a similar approach for guest_memfd HugeTLB, where perhaps the interface could be that userspace will give guest_memfd a HugeTLB fd at creation time, and then the original HugeTLB fd would be rendered unusable in the same way as above, perhaps like with the S_IMMUTABLE inode flag, but also blocking reads, and not userspace-modifiable.
In the course of a CoCo guest's operation, will the guest need to convert between private/shared MMIO? Will the guest need some pages shared and others private? If these are required operations, guest_memfd already provides the tracking and is going to have a conversion ioctl very soon. Instead of further extending dmabuf to track more things, how about letting guest_memfd track it?
On Mon, Jul 13, 2026 at 08:08:14PM +0100, Fuad Tabba wrote:
On Sun, 12 Jul 2026 at 02:02, Ackerley Tng ackerleytng@google.com wrote:
Jason Gunthorpe jgg@nvidia.com writes:
On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
I'd like to propose an alternative. I've been working on guest_memfd and new to the world of IO, please help me along! :)
It seems like using dmabufs are used a little awkwardly here. IIUC dmabufs were originally meant to expose memory of one device to another device, mostly meant to share memory. Dmabufs do expose MMIO too, for device to device communications. Without virtualization, userspace MMIO would be done by mmap()-ing a VFIO fd and having the userspace program write to the userspace addresses.
Before CoCo, device passthrough (MMIO) is mostly handled by mmap()-ing a VFIO fd and setting up the userspace address in a KVM memslot for the guest.
With CoCo, is the problem we're solving that we want KVM to know what pfns to set up in stage 2 page tables, but not via userspace addresses?
guest_memfd already does that for regular host memory, tracks the private/shared-ness of the memory, tracks which struct kvm the memory belongs to.
guest_memfd functions as KVM's bridge to host memory. KVM already can ask guest_memfd for the pfn to map into stage 2 page tables, and already asks guest_memfd for the shared/private state of the memory. guest_memfd already also blocks the host from faulting guest private memory (mmap()-ing is always allowed).
Instead of using dmabuf as the intermediary between the MMIO PFNs and KVM, why not use guest_memfd?
What if we make guest_memfd accept a VFIO fd, or a dmabuf fd?
This is interesting for pKVM too, provided it covers more than MMIO.
We need guest_memfd to be backable by a dmabuf for ordinary guest memory, not only for device MMIO. There is mobile hardware that doesn't tolerate scattered private memory (DMA engines that can't gather, IOMMU page-table size constraints), and a CMA-backed dmabuf heap is the practical way to get contiguous memory at runtime.
Why can't guestmemfd allocate directly from CMA? Allocating struct page memory through dmabuf just to put it back in a guestmemfd sounds very ugly to me.
HugeTLB doesn't help, it wants boot-time reservation. Those pages are struct-page backed, so it's a different problem from the non-struct-page MMIO case, and the shared parts still need to be GUP-able.
Isn't dmabuf pretty allergic to mmaping refcounted struct page backed memory since that wrecks its lifetime model?
More important for the API shape: conversions have to work on subsets of such a region, at page granularity. A pKVM guest doesn't know what backs its memory, so it will issue share/unshare hypercalls over arbitrary ranges of whatever it was given. If a dmabuf-backed guest_memfd can only be converted as a whole, we can't use it for memory, and the guest can't be taught to care.
More reasons not to involve DMABUF since guestmemfd already does all of this...
Jason
Jason Gunthorpe jgg@nvidia.com writes:
On Mon, Jul 13, 2026 at 08:08:14PM +0100, Fuad Tabba wrote:
On Sun, 12 Jul 2026 at 02:02, Ackerley Tng ackerleytng@google.com wrote:
Jason Gunthorpe jgg@nvidia.com writes:
On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
I'd like to propose an alternative. I've been working on guest_memfd and new to the world of IO, please help me along! :)
It seems like using dmabufs are used a little awkwardly here. IIUC dmabufs were originally meant to expose memory of one device to another device, mostly meant to share memory. Dmabufs do expose MMIO too, for device to device communications. Without virtualization, userspace MMIO would be done by mmap()-ing a VFIO fd and having the userspace program write to the userspace addresses.
Before CoCo, device passthrough (MMIO) is mostly handled by mmap()-ing a VFIO fd and setting up the userspace address in a KVM memslot for the guest.
With CoCo, is the problem we're solving that we want KVM to know what pfns to set up in stage 2 page tables, but not via userspace addresses?
guest_memfd already does that for regular host memory, tracks the private/shared-ness of the memory, tracks which struct kvm the memory belongs to.
guest_memfd functions as KVM's bridge to host memory. KVM already can ask guest_memfd for the pfn to map into stage 2 page tables, and already asks guest_memfd for the shared/private state of the memory. guest_memfd already also blocks the host from faulting guest private memory (mmap()-ing is always allowed).
Instead of using dmabuf as the intermediary between the MMIO PFNs and KVM, why not use guest_memfd?
What if we make guest_memfd accept a VFIO fd, or a dmabuf fd?
This is interesting for pKVM too, provided it covers more than MMIO.
We need guest_memfd to be backable by a dmabuf for ordinary guest memory, not only for device MMIO. There is mobile hardware that doesn't tolerate scattered private memory (DMA engines that can't gather, IOMMU page-table size constraints), and a CMA-backed dmabuf heap is the practical way to get contiguous memory at runtime.
Why can't guestmemfd allocate directly from CMA? Allocating struct page memory through dmabuf just to put it back in a guestmemfd sounds very ugly to me.
I'd like to understand this use case better too, is using CMA through dmabuf basically because CMA doesn't really have a direct userspace interface? As in, for HugeTLB there's fd = memfd_create(HUGETLB) and HugeTLBfs fds, but not CMA?
I did hope for this API shape to extend beyong VFIO and HugeTLB to anything that has an fd today, but Frank did bring up the counter point that not all memory has an fd.
HugeTLB doesn't help, it wants boot-time reservation. Those pages are struct-page backed, so it's a different problem from the non-struct-page MMIO case, and the shared parts still need to be GUP-able.
Isn't dmabuf pretty allergic to mmaping refcounted struct page backed memory since that wrecks its lifetime model?
More important for the API shape: conversions have to work on subsets of such a region, at page granularity. A pKVM guest doesn't know what backs its memory, so it will issue share/unshare hypercalls over arbitrary ranges of whatever it was given.
For my future reference, does this mean for the CMA case, some page splitting on conversion to shared (so each page can have its own refcount to track users) will be necessary? (and merging)
If a dmabuf-backed guest_memfd can only be converted as a whole, we can't use it for memory, and the guest can't be taught to care.
More reasons not to involve DMABUF since guestmemfd already does all of this...
Jason
On Mon, 13 Jul 2026 at 20:17, Jason Gunthorpe jgg@nvidia.com wrote:
On Mon, Jul 13, 2026 at 08:08:14PM +0100, Fuad Tabba wrote:
On Sun, 12 Jul 2026 at 02:02, Ackerley Tng ackerleytng@google.com wrote:
Jason Gunthorpe jgg@nvidia.com writes:
On Thu, May 29, 2025 at 01:34:53PM +0800, Xu Yilun wrote:
Export vfio dma-buf specific info by attaching vfio_dma_buf_data in struct dma_buf::priv. Provide a helper vfio_dma_buf_get_data() for importers to fetch these data. Exporters identify VFIO dma-buf by successfully getting these data.
VFIO dma-buf supports disabling host access to these exported MMIO regions when the device is converted to private. Exporters like KVM need to identify this type of dma-buf to decide if it is good to use. KVM only allows host unaccessible MMIO regions been mapped in private roots.
Export struct kvm * handler attached to the vfio device. This allows KVM to do another sanity check. MMIO should only be assigned to a CoCo VM if its owner device is already assigned to the same VM.
This doesn't seem right, it should be encapsulated into the standard DMABUF API in some way.
I'd like to propose an alternative. I've been working on guest_memfd and new to the world of IO, please help me along! :)
It seems like using dmabufs are used a little awkwardly here. IIUC dmabufs were originally meant to expose memory of one device to another device, mostly meant to share memory. Dmabufs do expose MMIO too, for device to device communications. Without virtualization, userspace MMIO would be done by mmap()-ing a VFIO fd and having the userspace program write to the userspace addresses.
Before CoCo, device passthrough (MMIO) is mostly handled by mmap()-ing a VFIO fd and setting up the userspace address in a KVM memslot for the guest.
With CoCo, is the problem we're solving that we want KVM to know what pfns to set up in stage 2 page tables, but not via userspace addresses?
guest_memfd already does that for regular host memory, tracks the private/shared-ness of the memory, tracks which struct kvm the memory belongs to.
guest_memfd functions as KVM's bridge to host memory. KVM already can ask guest_memfd for the pfn to map into stage 2 page tables, and already asks guest_memfd for the shared/private state of the memory. guest_memfd already also blocks the host from faulting guest private memory (mmap()-ing is always allowed).
Instead of using dmabuf as the intermediary between the MMIO PFNs and KVM, why not use guest_memfd?
What if we make guest_memfd accept a VFIO fd, or a dmabuf fd?
This is interesting for pKVM too, provided it covers more than MMIO.
We need guest_memfd to be backable by a dmabuf for ordinary guest memory, not only for device MMIO. There is mobile hardware that doesn't tolerate scattered private memory (DMA engines that can't gather, IOMMU page-table size constraints), and a CMA-backed dmabuf heap is the practical way to get contiguous memory at runtime.
Why can't guestmemfd allocate directly from CMA? Allocating struct page memory through dmabuf just to put it back in a guestmemfd sounds very ugly to me.
Fair, and I think you're right. If guest_memfd can allocate from CMA directly that covers what we need for contiguous guest memory, and it's cleaner than routing it through a dmabuf. It also keeps the shared pages struct-page backed and GUP-able, which the CMA heap's own mmap doesn't, since it sets VM_PFNMAP. So going through a dmabuf for plain guest RAM would have cost us the thing we need on the shared side anyway.
Consider the request to be for guest_memfd to be able to give us physically contiguous memory. dmabuf was the mechanism I assumed, not the requirement.
HugeTLB doesn't help, it wants boot-time reservation. Those pages are struct-page backed, so it's a different problem from the non-struct-page MMIO case, and the shared parts still need to be GUP-able.
Isn't dmabuf pretty allergic to mmaping refcounted struct page backed memory since that wrecks its lifetime model?
Yes, and that's the same point. Agreed.
More important for the API shape: conversions have to work on subsets of such a region, at page granularity. A pKVM guest doesn't know what backs its memory, so it will issue share/unshare hypercalls over arbitrary ranges of whatever it was given. If a dmabuf-backed guest_memfd can only be converted as a whole, we can't use it for memory, and the guest can't be taught to care.
More reasons not to involve DMABUF since guestmemfd already does all of this...
Also agreed for guest RAM.
Where I do still think a dmabuf is involved is the case where the buffer isn't guest_memfd's to allocate: it already belongs to another exporter, and the guest needs to see that same buffer. That's structurally what you're already handling for device memory rather than a separate guest_memfd-over-dmabuf path, so I don't think it argues for backing ordinary guest memory with a dmabuf.
Jason
linaro-mm-sig@lists.linaro.org
-
Ackerley Tng -
Fuad Tabba -
Jason Gunthorpe