Add GUEST_MEMFD_FLAG_NO_DIRECT_MAP flag for KVM_CREATE_GUEST_MEMFD() ioctl. When set, guest_memfd folios will be removed from the direct map after preparation, with direct map entries only restored when the folios are freed.

To ensure these folios do not end up in places where the kernel cannot deal with them, set AS_NO_DIRECT_MAP on the guest_memfd's struct address_space if GUEST_MEMFD_FLAG_NO_DIRECT_MAP is requested.

Add KVM_CAP_GUEST_MEMFD_NO_DIRECT_MAP to let userspace discover whether guest_memfd supports GUEST_MEMFD_FLAG_NO_DIRECT_MAP. Support depends on guest_memfd itself being supported, but also on whether linux supports manipulatomg the direct map at page granularity at all (possible most of the time, outliers being arm64 where its impossible if the direct map has been setup using hugepages, as arm64 cannot break these apart due to break-before-make semantics, and powerpc, which does not select ARCH_HAS_SET_DIRECT_MAP, though also doesn't support guest_memfd anyway).

Note that this flag causes removal of direct map entries for all guest_memfd folios independent of whether they are "shared" or "private" (although current guest_memfd only supports either all folios in the "shared" state, or all folios in the "private" state if GUEST_MEMFD_FLAG_MMAP is not set). The usecase for removing direct map entries of also the shared parts of guest_memfd are a special type of non-CoCo VM where, host userspace is trusted to have access to all of guest memory, but where Spectre-style transient execution attacks through the host kernel's direct map should still be mitigated. In this setup, KVM retains access to guest memory via userspace mappings of guest_memfd, which are reflected back into KVM's memslots via userspace_addr. This is needed for things like MMIO emulation on x86_64 to work.

Direct map entries are zapped right before guest or userspace mappings of gmem folios are set up, e.g. in kvm_gmem_fault_user_mapping() or kvm_gmem_get_pfn() [called from the KVM MMU code]. The only place where a gmem folio can be allocated without being mapped anywhere is kvm_gmem_populate(), where handling potential failures of direct map removal is not possible (by the time direct map removal is attempted, the folio is already marked as prepared, meaning attempting to re-try kvm_gmem_populate() would just result in -EEXIST without fixing up the direct map state). These folios are then removed form the direct map upon kvm_gmem_get_pfn(), e.g. when they are mapped into the guest later.

Signed-off-by: Patrick Roy roypat@amazon.co.uk --- Documentation/virt/kvm/api.rst | 5 +++ arch/arm64/include/asm/kvm_host.h | 12 ++++++ include/linux/kvm_host.h | 6 +++ include/uapi/linux/kvm.h | 2 + virt/kvm/guest_memfd.c | 61 ++++++++++++++++++++++++++++++- virt/kvm/kvm_main.c | 5 +++ 6 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index c17a87a0a5ac..b52c14d58798 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6418,6 +6418,11 @@ When the capability KVM_CAP_GUEST_MEMFD_MMAP is supported, the 'flags' field supports GUEST_MEMFD_FLAG_MMAP. Setting this flag on guest_memfd creation enables mmap() and faulting of guest_memfd memory to host userspace.

+When the capability KVM_CAP_GMEM_NO_DIRECT_MAP is supported, the 'flags' field +supports GUEST_MEMFG_FLAG_NO_DIRECT_MAP. Setting this flag makes the guest_memfd +instance behave similarly to memfd_secret, and unmaps the memory backing it from +the kernel's address space after allocation. + When the KVM MMU performs a PFN lookup to service a guest fault and the backing guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be consumed from guest_memfd, regardless of whether it is a shared or a private diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 2f2394cce24e..0bfd8e5fd9de 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -19,6 +19,7 @@ #include <linux/maple_tree.h> #include <linux/percpu.h> #include <linux/psci.h> +#include <linux/set_memory.h> #include <asm/arch_gicv3.h> #include <asm/barrier.h> #include <asm/cpufeature.h> @@ -1706,5 +1707,16 @@ void compute_fgu(struct kvm *kvm, enum fgt_group_id fgt); void get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg, u64 *res0, u64 *res1); void check_feature_map(void);

+#ifdef CONFIG_KVM_GUEST_MEMFD +static inline bool kvm_arch_gmem_supports_no_direct_map(void) +{ + /* + * Without FWB, direct map access is needed in kvm_pgtable_stage2_map(), + * as it calls dcache_clean_inval_poc(). + */ + return can_set_direct_map() && cpus_have_final_cap(ARM64_HAS_STAGE2_FWB); +} +#define kvm_arch_gmem_supports_no_direct_map kvm_arch_gmem_supports_no_direct_map +#endif /* CONFIG_KVM_GUEST_MEMFD */

#endif /* __ARM64_KVM_HOST_H__ */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1d0585616aa3..73a15cade54a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -731,6 +731,12 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm) bool kvm_arch_supports_gmem_mmap(struct kvm *kvm); #endif

+#ifdef CONFIG_KVM_GUEST_MEMFD +#ifndef kvm_arch_gmem_supports_no_direct_map +#define kvm_arch_gmem_supports_no_direct_map can_set_direct_map +#endif +#endif /* CONFIG_KVM_GUEST_MEMFD */ + #ifndef kvm_arch_has_readonly_mem static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm) { diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 6efa98a57ec1..33c8e8946019 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -963,6 +963,7 @@ struct kvm_enable_cap { #define KVM_CAP_RISCV_MP_STATE_RESET 242 #define KVM_CAP_ARM_CACHEABLE_PFNMAP_SUPPORTED 243 #define KVM_CAP_GUEST_MEMFD_MMAP 244 +#define KVM_CAP_GUEST_MEMFD_NO_DIRECT_MAP 245

struct kvm_irq_routing_irqchip { __u32 irqchip; @@ -1600,6 +1601,7 @@ struct kvm_memory_attributes {

#define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd) #define GUEST_MEMFD_FLAG_MMAP (1ULL << 0) +#define GUEST_MEMFD_FLAG_NO_DIRECT_MAP (1ULL << 1)

struct kvm_create_guest_memfd { __u64 size; diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 55b8d739779f..b7129c4868c5 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -4,6 +4,9 @@ #include <linux/kvm_host.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> +#include <linux/set_memory.h> + +#include <asm/tlbflush.h>

#include "kvm_mm.h"

@@ -42,6 +45,44 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo return 0; }

+#define KVM_GMEM_FOLIO_NO_DIRECT_MAP BIT(0) + +static bool kvm_gmem_folio_no_direct_map(struct folio *folio) +{ + return ((u64) folio->private) & KVM_GMEM_FOLIO_NO_DIRECT_MAP; +} + +static int kvm_gmem_folio_zap_direct_map(struct folio *folio) +{ + if (kvm_gmem_folio_no_direct_map(folio)) + return 0; + + int r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio), + false); + + if (!r) { + unsigned long addr = (unsigned long) folio_address(folio); + folio->private = (void *) ((u64) folio->private & KVM_GMEM_FOLIO_NO_DIRECT_MAP); + flush_tlb_kernel_range(addr, addr + folio_size(folio)); + } + + return r; +} + +static void kvm_gmem_folio_restore_direct_map(struct folio *folio) +{ + /* + * Direct map restoration cannot fail, as the only error condition + * for direct map manipulation is failure to allocate page tables + * when splitting huge pages, but this split would have already + * happened in set_direct_map_invalid_noflush() in kvm_gmem_folio_zap_direct_map(). + * Thus set_direct_map_valid_noflush() here only updates prot bits. + */ + if (kvm_gmem_folio_no_direct_map(folio)) + set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio), + true); +} + static inline void kvm_gmem_mark_prepared(struct folio *folio) { folio_mark_uptodate(folio); @@ -324,13 +365,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct folio *folio; vm_fault_t ret = VM_FAULT_LOCKED; + int err;

if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS;

folio = kvm_gmem_get_folio(inode, vmf->pgoff); if (IS_ERR(folio)) { - int err = PTR_ERR(folio); + err = PTR_ERR(folio);

if (err == -EAGAIN) return VM_FAULT_RETRY; @@ -348,6 +390,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) kvm_gmem_mark_prepared(folio); }

+ err = kvm_gmem_folio_zap_direct_map(folio); + + if (err) { + ret = vmf_error(err); + goto out_folio; + } + vmf->page = folio_file_page(folio, vmf->pgoff);

out_folio: @@ -435,6 +484,8 @@ static void kvm_gmem_free_folio(struct folio *folio) kvm_pfn_t pfn = page_to_pfn(page); int order = folio_order(folio);

+ kvm_gmem_folio_restore_direct_map(folio); + kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order)); }

@@ -499,6 +550,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) /* Unmovable mappings are supposed to be marked unevictable as well. */ WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));

+ if (flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP) + mapping_set_no_direct_map(inode->i_mapping); + kvm_get_kvm(kvm); gmem->kvm = kvm; xa_init(&gmem->bindings); @@ -523,6 +577,9 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) if (kvm_arch_supports_gmem_mmap(kvm)) valid_flags |= GUEST_MEMFD_FLAG_MMAP;

+ if (kvm_arch_gmem_supports_no_direct_map()) + valid_flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP; + if (flags & ~valid_flags) return -EINVAL;

@@ -687,6 +744,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!is_prepared) r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

+ kvm_gmem_folio_zap_direct_map(folio); + folio_unlock(folio);

if (!r) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 18f29ef93543..b5e702d95230 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -65,6 +65,7 @@ #include <trace/events/kvm.h>

#include <linux/kvm_dirty_ring.h> +#include <linux/set_memory.h>

/* Worst case buffer size needed for holding an integer. */ @@ -4916,6 +4917,10 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) return kvm_supported_mem_attributes(kvm); #endif #ifdef CONFIG_KVM_GUEST_MEMFD + case KVM_CAP_GUEST_MEMFD_NO_DIRECT_MAP: + if (!kvm_arch_gmem_supports_no_direct_map()) + return 0; + fallthrough; case KVM_CAP_GUEST_MEMFD: return 1; case KVM_CAP_GUEST_MEMFD_MMAP:

On Thu, 2025-09-25 at 12:00 +0100, David Hildenbrand wrote:

On 24.09.25 17:22, Roy, Patrick wrote:

...

Add GUEST_MEMFD_FLAG_NO_DIRECT_MAP flag for KVM_CREATE_GUEST_MEMFD() ioctl. When set, guest_memfd folios will be removed from the direct map after preparation, with direct map entries only restored when the folios are freed.

To ensure these folios do not end up in places where the kernel cannot deal with them, set AS_NO_DIRECT_MAP on the guest_memfd's struct address_space if GUEST_MEMFD_FLAG_NO_DIRECT_MAP is requested.

Add KVM_CAP_GUEST_MEMFD_NO_DIRECT_MAP to let userspace discover whether guest_memfd supports GUEST_MEMFD_FLAG_NO_DIRECT_MAP. Support depends on guest_memfd itself being supported, but also on whether linux supports manipulatomg the direct map at page granularity at all (possible most of the time, outliers being arm64 where its impossible if the direct map has been setup using hugepages, as arm64 cannot break these apart due to break-before-make semantics, and powerpc, which does not select ARCH_HAS_SET_DIRECT_MAP, though also doesn't support guest_memfd anyway).

Note that this flag causes removal of direct map entries for all guest_memfd folios independent of whether they are "shared" or "private" (although current guest_memfd only supports either all folios in the "shared" state, or all folios in the "private" state if GUEST_MEMFD_FLAG_MMAP is not set). The usecase for removing direct map entries of also the shared parts of guest_memfd are a special type of non-CoCo VM where, host userspace is trusted to have access to all of guest memory, but where Spectre-style transient execution attacks through the host kernel's direct map should still be mitigated. In this setup, KVM retains access to guest memory via userspace mappings of guest_memfd, which are reflected back into KVM's memslots via userspace_addr. This is needed for things like MMIO emulation on x86_64 to work.

Direct map entries are zapped right before guest or userspace mappings of gmem folios are set up, e.g. in kvm_gmem_fault_user_mapping() or kvm_gmem_get_pfn() [called from the KVM MMU code]. The only place where a gmem folio can be allocated without being mapped anywhere is kvm_gmem_populate(), where handling potential failures of direct map removal is not possible (by the time direct map removal is attempted, the folio is already marked as prepared, meaning attempting to re-try kvm_gmem_populate() would just result in -EEXIST without fixing up the direct map state). These folios are then removed form the direct map upon kvm_gmem_get_pfn(), e.g. when they are mapped into the guest later.

Signed-off-by: Patrick Roy roypat@amazon.co.uk

Documentation/virt/kvm/api.rst | 5 +++ arch/arm64/include/asm/kvm_host.h | 12 ++++++ include/linux/kvm_host.h | 6 +++ include/uapi/linux/kvm.h | 2 + virt/kvm/guest_memfd.c | 61 ++++++++++++++++++++++++++++++- virt/kvm/kvm_main.c | 5 +++ 6 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index c17a87a0a5ac..b52c14d58798 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6418,6 +6418,11 @@ When the capability KVM_CAP_GUEST_MEMFD_MMAP is supported, the 'flags' field supports GUEST_MEMFD_FLAG_MMAP. Setting this flag on guest_memfd creation enables mmap() and faulting of guest_memfd memory to host userspace.

+When the capability KVM_CAP_GMEM_NO_DIRECT_MAP is supported, the 'flags' field +supports GUEST_MEMFG_FLAG_NO_DIRECT_MAP. Setting this flag makes the guest_memfd +instance behave similarly to memfd_secret, and unmaps the memory backing it from +the kernel's address space after allocation.

Do we want to document what the implication of that is? Meaning, limitations etc. I recall that we would need the user mapping for gmem slots to be properly set up.

Is that still the case in this patch set?

The ->userspace_addr thing is the general requirement for non-CoCo VMs, and not specific for direct map removal (e.g. I expect direct map removal to just work out of the box for CoCo setups, where KVM already cannot access guest memory, ignoring the question of whether direct map removal is even useful for CoCo VMs). So I don't think it should be documented as part of KVM_CAP_GMEM_NO_DIRECT_MAP/GUEST_MEMFG_FLAG_NO_DIRECT_MAP (heh, there's a typo I just noticed. "MEMFG". Also "GMEM" needs to be "GUEST_MEMFD". Will fix that), but rather as part of GUEST_MEMFD_FLAG_MMAP. I can add a patch it there (or maybe send it separately, since FLAG_MMAP is already in -next?).

...

When the KVM MMU performs a PFN lookup to service a guest fault and the backing guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be consumed from guest_memfd, regardless of whether it is a shared or a private diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 2f2394cce24e..0bfd8e5fd9de 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -19,6 +19,7 @@ #include <linux/maple_tree.h> #include <linux/percpu.h> #include <linux/psci.h> +#include <linux/set_memory.h> #include <asm/arch_gicv3.h> #include <asm/barrier.h> #include <asm/cpufeature.h> @@ -1706,5 +1707,16 @@ void compute_fgu(struct kvm *kvm, enum fgt_group_id fgt); void get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg, u64 *res0, u64 *res1); void check_feature_map(void);

+#ifdef CONFIG_KVM_GUEST_MEMFD +static inline bool kvm_arch_gmem_supports_no_direct_map(void) +{

• /*
  

•  * Without FWB, direct map access is needed in kvm_pgtable_stage2_map(),
  

•  * as it calls dcache_clean_inval_poc().
  

•  */
  

• return can_set_direct_map() && cpus_have_final_cap(ARM64_HAS_STAGE2_FWB);
  

+} +#define kvm_arch_gmem_supports_no_direct_map kvm_arch_gmem_supports_no_direct_map +#endif /* CONFIG_KVM_GUEST_MEMFD */

I strongly assume that the aarch64 support should be moved to a separate patch -- if possible, see below.

...

#endif /* __ARM64_KVM_HOST_H__ */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1d0585616aa3..73a15cade54a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -731,6 +731,12 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm) bool kvm_arch_supports_gmem_mmap(struct kvm *kvm); #endif

+#ifdef CONFIG_KVM_GUEST_MEMFD +#ifndef kvm_arch_gmem_supports_no_direct_map +#define kvm_arch_gmem_supports_no_direct_map can_set_direct_map +#endif

Hm, wouldn't it be better to have an opt-in per arch, and really only unlock the ones we know work (tested etc), explicitly in separate patches.

Ack, can definitely do that. Something like

#ifndef kvm_arch_gmem_supports_no_direct_map static inline bool kvm_arch_gmem_supports_no_direct_map() { return false; } #endif

and then actual definitions (in separate patches) in the arm64 and x86 headers?

On a related note, maybe PATCH 2 should only export set_direct_map_valid_noflush() for the architectures on which we actually need it? Which would only be x86, since arm64 doesnt allow building KVM as a module, and nothing else supports guest_memfd right now.

[...]

...

#include "kvm_mm.h"

@@ -42,6 +45,44 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo return 0; }

+#define KVM_GMEM_FOLIO_NO_DIRECT_MAP BIT(0)

+static bool kvm_gmem_folio_no_direct_map(struct folio *folio) +{

• return ((u64) folio->private) & KVM_GMEM_FOLIO_NO_DIRECT_MAP;
  

+}

+static int kvm_gmem_folio_zap_direct_map(struct folio *folio) +{

• if (kvm_gmem_folio_no_direct_map(folio))
  

•         return 0;
  

• int r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),
  

•                                  false);
  

• if (!r) {
  

•         unsigned long addr = (unsigned long) folio_address(folio);
  

empty line missing.

Ack

...

•         folio->private = (void *) ((u64) folio->private & KVM_GMEM_FOLIO_NO_DIRECT_MAP);
  

•         flush_tlb_kernel_range(addr, addr + folio_size(folio));
  

• }
  

• return r;
  

+}

+static void kvm_gmem_folio_restore_direct_map(struct folio *folio) +{

• /*
  

•  * Direct map restoration cannot fail, as the only error condition
  

•  * for direct map manipulation is failure to allocate page tables
  

•  * when splitting huge pages, but this split would have already
  

•  * happened in set_direct_map_invalid_noflush() in kvm_gmem_folio_zap_direct_map().
  

•  * Thus set_direct_map_valid_noflush() here only updates prot bits.
  

•  */
  

• if (kvm_gmem_folio_no_direct_map(folio))
  

•         set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),
  

•                                  true);
  

+}

• static inline void kvm_gmem_mark_prepared(struct folio *folio) { folio_mark_uptodate(folio);

@@ -324,13 +365,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct folio *folio; vm_fault_t ret = VM_FAULT_LOCKED;

• int err;
  
  if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode))
          return VM_FAULT_SIGBUS;
  
  folio = kvm_gmem_get_folio(inode, vmf->pgoff);
  if (IS_ERR(folio)) {
  

•         int err = PTR_ERR(folio);
  

•         err = PTR_ERR(folio);
  
          if (err == -EAGAIN)
                  return VM_FAULT_RETRY;
  

@@ -348,6 +390,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) kvm_gmem_mark_prepared(folio); }

• err = kvm_gmem_folio_zap_direct_map(folio);
  

I'd drop this empty line here.

Ack

...

• if (err) {
  

•         ret = vmf_error(err);
  

•         goto out_folio;
  

• }
  

• vmf->page = folio_file_page(folio, vmf->pgoff);
  
  

  out_folio:

@@ -435,6 +484,8 @@ static void kvm_gmem_free_folio(struct folio *folio) kvm_pfn_t pfn = page_to_pfn(page); int order = folio_order(folio);

• kvm_gmem_folio_restore_direct_map(folio);
  

• kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
  

  }

@@ -499,6 +550,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) /* Unmovable mappings are supposed to be marked unevictable as well. */ WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));

• if (flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP)
  

•         mapping_set_no_direct_map(inode->i_mapping);
  

• kvm_get_kvm(kvm);
  gmem->kvm = kvm;
  xa_init(&gmem->bindings);
  

@@ -523,6 +577,9 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) if (kvm_arch_supports_gmem_mmap(kvm)) valid_flags |= GUEST_MEMFD_FLAG_MMAP;

• if (kvm_arch_gmem_supports_no_direct_map())
  

•         valid_flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP;
  

• if (flags & ~valid_flags)
          return -EINVAL;
  
  

@@ -687,6 +744,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!is_prepared) r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

• kvm_gmem_folio_zap_direct_map(folio);
  

• folio_unlock(folio);
  
  if (!r)
  

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 18f29ef93543..b5e702d95230 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -65,6 +65,7 @@ #include <trace/events/kvm.h>

#include <linux/kvm_dirty_ring.h> +#include <linux/set_memory.h>

Likely not required here.

Seems for now it is because of how kvm_arch_gmem_supports_no_direct_map() is defined, but I suspect the need will disappear once that is changed as you suggested above :)

-- Cheers

David / dhildenb

Best, Patrick

On Wed, 2025-09-24 at 16:22 +0100, "Roy, Patrick" wrote:

[...]

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 55b8d739779f..b7129c4868c5 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -4,6 +4,9 @@ #include <linux/kvm_host.h> #include <linux/pagemap.h> #include <linux/anon_inodes.h> +#include <linux/set_memory.h>

+#include <asm/tlbflush.h> #include "kvm_mm.h" @@ -42,6 +45,44 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo return 0; } +#define KVM_GMEM_FOLIO_NO_DIRECT_MAP BIT(0)

+static bool kvm_gmem_folio_no_direct_map(struct folio *folio) +{

• return ((u64) folio->private) & KVM_GMEM_FOLIO_NO_DIRECT_MAP;

+}

+static int kvm_gmem_folio_zap_direct_map(struct folio *folio) +{

• if (kvm_gmem_folio_no_direct_map(folio))

• return 0;
  

• int r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),

• 			 false);
  

• if (!r) {

• unsigned long addr = (unsigned long) folio_address(folio);
  

• folio->private = (void *) ((u64) folio->private & KVM_GMEM_FOLIO_NO_DIRECT_MAP);
  

• flush_tlb_kernel_range(addr, addr + folio_size(folio));
  

• }
• return r;

+}

No idea how I managed to mess this function up so completely, but it should be more like

static int kvm_gmem_folio_zap_direct_map(struct folio *folio) { int r = 0; unsigned long addr = (unsigned long) folio_address(folio); u64 gmem_flags = (u64) folio_inode(folio)->i_private;

if (kvm_gmem_folio_no_direct_map(folio) || !(gmem_flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP)) goto out;

r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio), false);

if (r) goto out;

folio->private = (void *) KVM_GMEM_FOLIO_NO_DIRECT_MAP; flush_tlb_kernel_range(addr, addr + folio_size(folio));

out: return r; }

the version I sent (a) does not respect the flags passed to guest_memfd on creation, and (b) does not correctly set the bit in folio->private.

+static void kvm_gmem_folio_restore_direct_map(struct folio *folio) +{

• /*

• * Direct map restoration cannot fail, as the only error condition
  

• * for direct map manipulation is failure to allocate page tables
  

• * when splitting huge pages, but this split would have already
  

• * happened in set_direct_map_invalid_noflush() in kvm_gmem_folio_zap_direct_map().
  

• * Thus set_direct_map_valid_noflush() here only updates prot bits.
  

• */
  

• if (kvm_gmem_folio_no_direct_map(folio))

• set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),
  

• 			 true);
  

+}

static inline void kvm_gmem_mark_prepared(struct folio *folio) { folio_mark_uptodate(folio); @@ -324,13 +365,14 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct folio *folio; vm_fault_t ret = VM_FAULT_LOCKED;

• int err;

if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; folio = kvm_gmem_get_folio(inode, vmf->pgoff); if (IS_ERR(folio)) {

• int err = PTR_ERR(folio);
  

• err = PTR_ERR(folio);
  

if (err == -EAGAIN) return VM_FAULT_RETRY; @@ -348,6 +390,13 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) kvm_gmem_mark_prepared(folio); }

• err = kvm_gmem_folio_zap_direct_map(folio);
• if (err) {

• ret = vmf_error(err);
  

• goto out_folio;
  

• }
• vmf->page = folio_file_page(folio, vmf->pgoff);

out_folio: @@ -435,6 +484,8 @@ static void kvm_gmem_free_folio(struct folio *folio) kvm_pfn_t pfn = page_to_pfn(page); int order = folio_order(folio);

• kvm_gmem_folio_restore_direct_map(folio);
• kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));

} @@ -499,6 +550,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) /* Unmovable mappings are supposed to be marked unevictable as well. */ WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));

• if (flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP)

• mapping_set_no_direct_map(inode->i_mapping);
  

• kvm_get_kvm(kvm); gmem->kvm = kvm; xa_init(&gmem->bindings);

@@ -523,6 +577,9 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) if (kvm_arch_supports_gmem_mmap(kvm)) valid_flags |= GUEST_MEMFD_FLAG_MMAP;

• if (kvm_arch_gmem_supports_no_direct_map())

• valid_flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP;
  

• if (flags & ~valid_flags) return -EINVAL;

@@ -687,6 +744,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!is_prepared) r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

• kvm_gmem_folio_zap_direct_map(folio);
• folio_unlock(folio);

if (!r)

[...]

On Fri, Oct 31, 2025 at 05:30:12PM +0000, Brendan Jackman wrote:

On Wed Sep 24, 2025 at 3:22 PM UTC, Patrick Roy wrote:

...

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1d0585616aa3..73a15cade54a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -731,6 +731,12 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm) bool kvm_arch_supports_gmem_mmap(struct kvm *kvm); #endif +#ifdef CONFIG_KVM_GUEST_MEMFD +#ifndef kvm_arch_gmem_supports_no_direct_map +#define kvm_arch_gmem_supports_no_direct_map can_set_direct_map +#endif +#endif /* CONFIG_KVM_GUEST_MEMFD */

The test robot seems happy so I think I'm probably mistaken here, but AFAICS can_set_direct_map only exists when ARCH_HAS_SET_DIRECT_MAP, which powerpc doesn't set.

We have stubs returning 0 for architectures that don't have ARCH_HAS_SET_DIRECT_MAP.

If this is indeed an issue I think it can be fixed by just defining can_set_direct_map() to false when !ARCH_HAS_SET_DIRECT_MAP.

On Mon, Nov 03, 2025 at 10:35:38AM +0000, Brendan Jackman wrote:

On Sat Nov 1, 2025 at 9:39 AM UTC, Mike Rapoport wrote:

...

On Fri, Oct 31, 2025 at 05:30:12PM +0000, Brendan Jackman wrote:

...

On Wed Sep 24, 2025 at 3:22 PM UTC, Patrick Roy wrote:

...

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1d0585616aa3..73a15cade54a 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -731,6 +731,12 @@ static inline bool kvm_arch_has_private_mem(struct kvm *kvm) bool kvm_arch_supports_gmem_mmap(struct kvm *kvm); #endif +#ifdef CONFIG_KVM_GUEST_MEMFD +#ifndef kvm_arch_gmem_supports_no_direct_map +#define kvm_arch_gmem_supports_no_direct_map can_set_direct_map +#endif +#endif /* CONFIG_KVM_GUEST_MEMFD */

The test robot seems happy so I think I'm probably mistaken here, but AFAICS can_set_direct_map only exists when ARCH_HAS_SET_DIRECT_MAP, which powerpc doesn't set.

We have stubs returning 0 for architectures that don't have ARCH_HAS_SET_DIRECT_MAP.

I can't see any such stub for can_set_direct_map() specifically?

include/linux/set_memory.h:

#ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP static inline int set_direct_map_invalid_noflush(struct page *page) { return 0; } static inline int set_direct_map_default_noflush(struct page *page) { return 0; }

static inline int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) { return 0; }

static inline bool kernel_page_present(struct page *page) { return true; } #else /* CONFIG_ARCH_HAS_SET_DIRECT_MAP */ /* * Some architectures, e.g. ARM64 can disable direct map modifications at * boot time. Let them overrive this query. */ #ifndef can_set_direct_map static inline bool can_set_direct_map(void) { return true; } #define can_set_direct_map can_set_direct_map #endif #endif /* CONFIG_ARCH_HAS_SET_DIRECT_MAP */

(But again, the bot seems happy, so I still suspect I'm wrong somehow or other).

Add an option to not perform TLB flushes after direct map manipulations. TLB flushes result in a up to 40x elongation of page faults in guest_memfd (scaling with the number of CPU cores), or a 5x elongation of memory population, which is inacceptable when wanting to use direct map removed guest_memfd as a drop-in replacement for existing workloads.

TLB flushes are not needed for functional correctness (the virt->phys mapping technically stays "correct", the kernel should simply not use it for a while), so we can skip them to keep performance in-line with "traditional" VMs.

Enabling this option means that the desired protection from Spectre-style attacks is not perfect, as an attacker could try to prevent a stale TLB entry from getting evicted, keeping it alive until the page it refers to is used by the guest for some sensitive data, and then targeting it using a spectre-gadget.

Cc: Will Deacon will@kernel.org Signed-off-by: Patrick Roy roypat@amazon.co.uk --- include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 3 ++- virt/kvm/kvm_main.c | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 73a15cade54a..4d2bc18860fc 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2298,6 +2298,7 @@ extern unsigned int halt_poll_ns; extern unsigned int halt_poll_ns_grow; extern unsigned int halt_poll_ns_grow_start; extern unsigned int halt_poll_ns_shrink; +extern bool guest_memfd_tlb_flush;

struct kvm_device { const struct kvm_device_ops *ops; diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index b7129c4868c5..d8dd24459f0d 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -63,7 +63,8 @@ static int kvm_gmem_folio_zap_direct_map(struct folio *folio) if (!r) { unsigned long addr = (unsigned long) folio_address(folio); folio->private = (void *) ((u64) folio->private & KVM_GMEM_FOLIO_NO_DIRECT_MAP); - flush_tlb_kernel_range(addr, addr + folio_size(folio)); + if (guest_memfd_tlb_flush) + flush_tlb_kernel_range(addr, addr + folio_size(folio)); }

return r; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index b5e702d95230..753c06ebba7f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -95,6 +95,9 @@ unsigned int halt_poll_ns_shrink = 2; module_param(halt_poll_ns_shrink, uint, 0644); EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);

+bool guest_memfd_tlb_flush = true; +module_param(guest_memfd_tlb_flush, bool, 0444); + /* * Allow direct access (from KVM or the CPU) without MMU notifier protection * to unpinned pages.

On Thu, 2025-09-25 at 12:02 +0100, David Hildenbrand wrote:

On 24.09.25 17:22, Roy, Patrick wrote:

...

Add an option to not perform TLB flushes after direct map manipulations. TLB flushes result in a up to 40x elongation of page faults in guest_memfd (scaling with the number of CPU cores), or a 5x elongation of memory population, which is inacceptable when wanting to use direct map removed guest_memfd as a drop-in replacement for existing workloads.

TLB flushes are not needed for functional correctness (the virt->phys mapping technically stays "correct", the kernel should simply not use it for a while), so we can skip them to keep performance in-line with "traditional" VMs.

Enabling this option means that the desired protection from Spectre-style attacks is not perfect, as an attacker could try to prevent a stale TLB entry from getting evicted, keeping it alive until the page it refers to is used by the guest for some sensitive data, and then targeting it using a spectre-gadget.

Cc: Will Deacon will@kernel.org Signed-off-by: Patrick Roy roypat@amazon.co.uk

---

include/linux/kvm_host.h | 1 + virt/kvm/guest_memfd.c | 3 ++- virt/kvm/kvm_main.c | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 73a15cade54a..4d2bc18860fc 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -2298,6 +2298,7 @@ extern unsigned int halt_poll_ns; extern unsigned int halt_poll_ns_grow; extern unsigned int halt_poll_ns_grow_start; extern unsigned int halt_poll_ns_shrink; +extern bool guest_memfd_tlb_flush;

struct kvm_device { const struct kvm_device_ops *ops; diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index b7129c4868c5..d8dd24459f0d 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -63,7 +63,8 @@ static int kvm_gmem_folio_zap_direct_map(struct folio *folio) if (!r) { unsigned long addr = (unsigned long) folio_address(folio); folio->private = (void *) ((u64) folio->private & KVM_GMEM_FOLIO_NO_DIRECT_MAP);

•         flush_tlb_kernel_range(addr, addr + folio_size(folio));
  

•         if (guest_memfd_tlb_flush)
  

•                 flush_tlb_kernel_range(addr, addr + folio_size(folio));
  }
  
  return r;
  

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index b5e702d95230..753c06ebba7f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -95,6 +95,9 @@ unsigned int halt_poll_ns_shrink = 2; module_param(halt_poll_ns_shrink, uint, 0644); EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);

+bool guest_memfd_tlb_flush = true; +module_param(guest_memfd_tlb_flush, bool, 0444);

The parameter name is a bit too generic. I think you somehow have to incorporate the "direct_map" aspects.

Fair :)

Also, I wonder if this could be a capability per vm/guest_memfd?

I don't really have any opinions on how to expose this knob, but I thought capabilities should be additive? (e.g. we only have KVM_ENABLE_EXTENSION(), and then having a capability with a negative polarity "enable to _not_ do TLB flushes" is a bit weird in my head). Then again, if people are fine having TLB flushes be opt-in instead of opt-out (Will's comment on v6 makes me believe that the opt-out itself might already be controversial for arm64), a capability would work.

Then, you could also nicely document the semantics, considerations, impact etc :)

Yup, I got so lost in trying to figure out why flush_kernel_tlb_range() didnt refused to let itself be exported that docs slipped my mind haha.

-- Cheers

David / dhildenb

Best, Patrick

If guest memory is backed using a VMA that does not allow GUP (e.g. a userspace mapping of guest_memfd when the fd was allocated using KVM_GMEM_NO_DIRECT_MAP), then directly loading the test ELF binary into it via read(2) potentially does not work. To nevertheless support loading binaries in this cases, do the read(2) syscall using a bounce buffer, and then memcpy from the bounce buffer into guest memory.

Signed-off-by: Patrick Roy roypat@amazon.co.uk --- .../testing/selftests/kvm/include/test_util.h | 1 + tools/testing/selftests/kvm/lib/elf.c | 8 +++---- tools/testing/selftests/kvm/lib/io.c | 23 +++++++++++++++++++ 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index c6ef895fbd9a..0409b7b96c94 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -46,6 +46,7 @@ do { \

ssize_t test_write(int fd, const void *buf, size_t count); ssize_t test_read(int fd, void *buf, size_t count); +ssize_t test_read_bounce(int fd, void *buf, size_t count); int test_seq_read(const char *path, char **bufp, size_t *sizep);

void __printf(5, 6) test_assert(bool exp, const char *exp_str, diff --git a/tools/testing/selftests/kvm/lib/elf.c b/tools/testing/selftests/kvm/lib/elf.c index f34d926d9735..e829fbe0a11e 100644 --- a/tools/testing/selftests/kvm/lib/elf.c +++ b/tools/testing/selftests/kvm/lib/elf.c @@ -31,7 +31,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) * the real size of the ELF header. */ unsigned char ident[EI_NIDENT]; - test_read(fd, ident, sizeof(ident)); + test_read_bounce(fd, ident, sizeof(ident)); TEST_ASSERT((ident[EI_MAG0] == ELFMAG0) && (ident[EI_MAG1] == ELFMAG1) && (ident[EI_MAG2] == ELFMAG2) && (ident[EI_MAG3] == ELFMAG3), "ELF MAGIC Mismatch,\n" @@ -79,7 +79,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) offset_rv = lseek(fd, 0, SEEK_SET); TEST_ASSERT(offset_rv == 0, "Seek to ELF header failed,\n" " rv: %zi expected: %i", offset_rv, 0); - test_read(fd, hdrp, sizeof(*hdrp)); + test_read_bounce(fd, hdrp, sizeof(*hdrp)); TEST_ASSERT(hdrp->e_phentsize == sizeof(Elf64_Phdr), "Unexpected physical header size,\n" " hdrp->e_phentsize: %x\n" @@ -146,7 +146,7 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename)

/* Read in the program header. */ Elf64_Phdr phdr; - test_read(fd, &phdr, sizeof(phdr)); + test_read_bounce(fd, &phdr, sizeof(phdr));

/* Skip if this header doesn't describe a loadable segment. */ if (phdr.p_type != PT_LOAD) @@ -187,7 +187,7 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename) " expected: 0x%jx", n1, errno, (intmax_t) offset_rv, (intmax_t) phdr.p_offset); - test_read(fd, addr_gva2hva(vm, phdr.p_vaddr), + test_read_bounce(fd, addr_gva2hva(vm, phdr.p_vaddr), phdr.p_filesz); } } diff --git a/tools/testing/selftests/kvm/lib/io.c b/tools/testing/selftests/kvm/lib/io.c index fedb2a741f0b..74419becc8bc 100644 --- a/tools/testing/selftests/kvm/lib/io.c +++ b/tools/testing/selftests/kvm/lib/io.c @@ -155,3 +155,26 @@ ssize_t test_read(int fd, void *buf, size_t count)

return num_read; } + +/* Test read via intermediary buffer + * + * Same as test_read, except read(2)s happen into a bounce buffer that is memcpy'd + * to buf. For use with buffers that cannot be GUP'd (e.g. guest_memfd VMAs if + * guest_memfd was created with GUEST_MEMFD_FLAG_NO_DIRECT_MAP). + */ +ssize_t test_read_bounce(int fd, void *buf, size_t count) +{ + void *bounce_buffer; + ssize_t num_read; + + TEST_ASSERT(count >= 0, "Unexpected count, count: %li", count); + + bounce_buffer = malloc(count); + TEST_ASSERT(bounce_buffer != NULL, "Failed to allocate bounce buffer"); + + num_read = test_read(fd, bounce_buffer, count); + memcpy(buf, bounce_buffer, num_read); + free(bounce_buffer); + + return num_read; +}

Allow selftests to configure their memslots such that userspace_addr is set to a MAP_SHARED mapping of the guest_memfd that's associated with the memslot. This setup is the configuration for non-CoCo VMs, where all guest memory is backed by a guest_memfd whose folios are all marked shared, but KVM is still able to access guest memory to provide functionality such as MMIO emulation on x86.

Add backing types for normal guest_memfd, as well as direct map removed guest_memfd.

Signed-off-by: Patrick Roy roypat@amazon.co.uk --- .../testing/selftests/kvm/include/kvm_util.h | 18 ++++++ .../testing/selftests/kvm/include/test_util.h | 7 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 63 ++++++++++--------- tools/testing/selftests/kvm/lib/test_util.c | 8 +++ 4 files changed, 66 insertions(+), 30 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 23a506d7eca3..5204a0a18a7f 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -635,6 +635,24 @@ static inline bool is_smt_on(void)

void vm_create_irqchip(struct kvm_vm *vm);

+static inline uint32_t backing_src_guest_memfd_flags(enum vm_mem_backing_src_type t) +{ + uint32_t flags = 0; + + switch (t) { + case VM_MEM_SRC_GUEST_MEMFD: + flags |= GUEST_MEMFD_FLAG_MMAP; + fallthrough; + case VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP: + flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP; + break; + default: + break; + } + + return flags; +} + static inline int __vm_create_guest_memfd(struct kvm_vm *vm, uint64_t size, uint64_t flags) { diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index 0409b7b96c94..a56e53fc7b39 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -133,6 +133,8 @@ enum vm_mem_backing_src_type { VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB, VM_MEM_SRC_SHMEM, VM_MEM_SRC_SHARED_HUGETLB, + VM_MEM_SRC_GUEST_MEMFD, + VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP, NUM_SRC_TYPES, };

@@ -165,6 +167,11 @@ static inline bool backing_src_is_shared(enum vm_mem_backing_src_type t) return vm_mem_backing_src_alias(t)->flag & MAP_SHARED; }

+static inline bool backing_src_is_guest_memfd(enum vm_mem_backing_src_type t) +{ + return t == VM_MEM_SRC_GUEST_MEMFD || t == VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP; +} + static inline bool backing_src_can_be_huge(enum vm_mem_backing_src_type t) { return t != VM_MEM_SRC_ANONYMOUS && t != VM_MEM_SRC_SHMEM; diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index cc67dfecbf65..a81089f7c83f 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1060,6 +1060,34 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, alignment = 1; #endif

+ if (guest_memfd < 0) { + if ((flags & KVM_MEM_GUEST_MEMFD) || backing_src_is_guest_memfd(src_type)) { + uint32_t guest_memfd_flags = backing_src_guest_memfd_flags(src_type); + + TEST_ASSERT(!guest_memfd_offset, + "Offset must be zero when creating new guest_memfd"); + guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); + } + } else { + /* + * Install a unique fd for each memslot so that the fd + * can be closed when the region is deleted without + * needing to track if the fd is owned by the framework + * or by the caller. + */ + guest_memfd = dup(guest_memfd); + TEST_ASSERT(guest_memfd >= 0, __KVM_SYSCALL_ERROR("dup()", guest_memfd)); + } + + if (guest_memfd > 0) { + flags |= KVM_MEM_GUEST_MEMFD; + + region->region.guest_memfd = guest_memfd; + region->region.guest_memfd_offset = guest_memfd_offset; + } else { + region->region.guest_memfd = -1; + } + /* * When using THP mmap is not guaranteed to returned a hugepage aligned * address so we have to pad the mmap. Padding is not needed for HugeTLB @@ -1075,10 +1103,13 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, if (alignment > 1) region->mmap_size += alignment;

- region->fd = -1; - if (backing_src_is_shared(src_type)) + if (backing_src_is_guest_memfd(src_type)) + region->fd = guest_memfd; + else if (backing_src_is_shared(src_type)) region->fd = kvm_memfd_alloc(region->mmap_size, src_type == VM_MEM_SRC_SHARED_HUGETLB); + else + region->fd = -1;

region->mmap_start = mmap(NULL, region->mmap_size, PROT_READ | PROT_WRITE, @@ -1106,34 +1137,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, }

region->backing_src_type = src_type; - - if (guest_memfd < 0) { - if (flags & KVM_MEM_GUEST_MEMFD) { - uint32_t guest_memfd_flags = 0; - TEST_ASSERT(!guest_memfd_offset, - "Offset must be zero when creating new guest_memfd"); - guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); - } - } else { - /* - * Install a unique fd for each memslot so that the fd - * can be closed when the region is deleted without - * needing to track if the fd is owned by the framework - * or by the caller. - */ - guest_memfd = dup(guest_memfd); - TEST_ASSERT(guest_memfd >= 0, __KVM_SYSCALL_ERROR("dup()", guest_memfd)); - } - - if (guest_memfd > 0) { - flags |= KVM_MEM_GUEST_MEMFD; - - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; - } else { - region->region.guest_memfd = -1; - } - region->unused_phy_pages = sparsebit_alloc(); if (vm_arch_has_protected_memory(vm)) region->protected_phy_pages = sparsebit_alloc(); diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index 03eb99af9b8d..b2baee680083 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -299,6 +299,14 @@ const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i) */ .flag = MAP_SHARED, }, + [VM_MEM_SRC_GUEST_MEMFD] = { + .name = "guest_memfd", + .flag = MAP_SHARED, + }, + [VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP] = { + .name = "guest_memfd_no_direct_map", + .flag = MAP_SHARED, + } }; _Static_assert(ARRAY_SIZE(aliases) == NUM_SRC_TYPES, "Missing new backing src types?");

Use one of the padding fields in struct vm_shape to carry an enum vm_mem_backing_src_type value, to give the option to overwrite the default of VM_MEM_SRC_ANONYMOUS in __vm_create().

Overwriting this default will allow tests to create VMs where the test code is backed by mmap'd guest_memfd instead of anonymous memory.

Signed-off-by: Patrick Roy roypat@amazon.co.uk --- .../testing/selftests/kvm/include/kvm_util.h | 19 ++++++++++--------- tools/testing/selftests/kvm/lib/kvm_util.c | 2 +- tools/testing/selftests/kvm/lib/x86/sev.c | 1 + .../selftests/kvm/pre_fault_memory_test.c | 1 + 4 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 5204a0a18a7f..8baa0bbacd09 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -188,7 +188,7 @@ enum vm_guest_mode { struct vm_shape { uint32_t type; uint8_t mode; - uint8_t pad0; + uint8_t src_type; uint16_t pad1; };

@@ -196,14 +196,15 @@ kvm_static_assert(sizeof(struct vm_shape) == sizeof(uint64_t));

#define VM_TYPE_DEFAULT 0

-#define VM_SHAPE(__mode) \ -({ \ - struct vm_shape shape = { \ - .mode = (__mode), \ - .type = VM_TYPE_DEFAULT \ - }; \ - \ - shape; \ +#define VM_SHAPE(__mode) \ +({ \ + struct vm_shape shape = { \ + .mode = (__mode), \ + .type = VM_TYPE_DEFAULT, \ + .src_type = VM_MEM_SRC_ANONYMOUS \ + }; \ + \ + shape; \ })

#if defined(__aarch64__) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index a81089f7c83f..3a22794bd959 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -495,7 +495,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, uint32_t nr_runnable_vcpus, if (is_guest_memfd_required(shape)) flags |= KVM_MEM_GUEST_MEMFD;

- vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, 0, 0, nr_pages, flags); + vm_userspace_mem_region_add(vm, shape.src_type, 0, 0, nr_pages, flags); for (i = 0; i < NR_MEM_REGIONS; i++) vm->memslots[i] = 0;

diff --git a/tools/testing/selftests/kvm/lib/x86/sev.c b/tools/testing/selftests/kvm/lib/x86/sev.c index c3a9838f4806..d920880e4fc0 100644 --- a/tools/testing/selftests/kvm/lib/x86/sev.c +++ b/tools/testing/selftests/kvm/lib/x86/sev.c @@ -164,6 +164,7 @@ struct kvm_vm *vm_sev_create_with_one_vcpu(uint32_t type, void *guest_code, struct vm_shape shape = { .mode = VM_MODE_DEFAULT, .type = type, + .src_type = VM_MEM_SRC_ANONYMOUS, }; struct kvm_vm *vm; struct kvm_vcpu *cpus[1]; diff --git a/tools/testing/selftests/kvm/pre_fault_memory_test.c b/tools/testing/selftests/kvm/pre_fault_memory_test.c index 0350a8896a2f..d403f8d2f26f 100644 --- a/tools/testing/selftests/kvm/pre_fault_memory_test.c +++ b/tools/testing/selftests/kvm/pre_fault_memory_test.c @@ -68,6 +68,7 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) const struct vm_shape shape = { .mode = VM_MODE_DEFAULT, .type = vm_type, + .src_type = VM_MEM_SRC_ANONYMOUS, }; struct kvm_vcpu *vcpu; struct kvm_run *run;

On Wed Sep 24, 2025 at 3:22 PM UTC, Patrick Roy wrote:

Add a selftest that loads itself into guest_memfd (via GUEST_MEMFD_FLAG_MMAP) and triggers an MMIO exit when executed. This exercises x86 MMIO emulation code inside KVM for guest_memfd-backed memslots where the guest_memfd folios are direct map removed. Particularly, it validates that x86 MMIO emulation code (guest page table walks + instruction fetch) correctly accesses gmem through the VMA that's been reflected into the memslot's userspace_addr field (instead of trying to do direct map accesses).

Signed-off-by: Patrick Roy roypat@amazon.co.uk

.../selftests/kvm/set_memory_region_test.c | 50 +++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/kvm/set_memory_region_test.c b/tools/testing/selftests/kvm/set_memory_region_test.c index ce3ac0fd6dfb..cb3bc642d376 100644 --- a/tools/testing/selftests/kvm/set_memory_region_test.c +++ b/tools/testing/selftests/kvm/set_memory_region_test.c @@ -603,6 +603,41 @@ static void test_mmio_during_vectoring(void) kvm_vm_free(vm); }

+static void guest_code_trigger_mmio(void) +{

• /*

• * Read some GPA that is not backed by a memslot. KVM consider this
  

• * as MMIO and tell userspace to emulate the read.
  

• */
  

• READ_ONCE(*((uint64_t *)MEM_REGION_GPA));
• GUEST_DONE();

+}

+static void test_guest_memfd_mmio(void) +{

• struct kvm_vm *vm;
• struct kvm_vcpu *vcpu;
• struct vm_shape shape = {

• .mode = VM_MODE_DEFAULT,
  

• .src_type = VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP,
  

• };
• pthread_t vcpu_thread;
• pr_info("Testing MMIO emulation for instructions in gmem\n");
• vm = __vm_create_shape_with_one_vcpu(shape, &vcpu, 0, guest_code_trigger_mmio);

When I run this test on my minimal config in a nested VM I get:

[root@testvm:~]# /nix/store/xlxd60n7v1qfr6s5zxda410zrzdd0xc2-kselftests/bin/run_kselftest.sh -t kvm:set_memory_region_test TAP version 13 1..1 # timeout set to 120 # selftests: kvm: set_memory_region_test # Random seed: 0x6b8b4567 # Testing KVM_RUN with zero added memory regions # Testing MMIO during vectoring error handling # Allowed number of memory slots: 32764 # Adding slots 0..32763, each memory region with 2048K size # Testing MMIO emulation for instructions in gmem # ==== Test Assertion Failure ==== # lib/kvm_util.c:1118: region->mmap_start != MAP_FAILED # pid=614 tid=614 errno=19 - No such device # 1 0x0000000000407b02: vm_mem_add at ??:? # 2 0x000000000040a924: __vm_create at ??:? # 3 0x000000000040ab16: __vm_create_shape_with_one_vcpu at ??:? # 4 0x00000000004042cf: main at ??:? # 5 0x00007faa6b08a47d: ?? ??:0 # 6 0x00007faa6b08a538: ?? ??:0 # 7 0x0000000000404384: _start at ??:? # mmap() failed, rc: -1 errno: 19 (No such device)

Here's the kconfig I'm using (basically defconfig+KVM):

https://gist.githubusercontent.com/bjackman/4ea941ef5072606769211f3b000c8ed7...

Sorry I'm too ignorant about KVM to know what I'm missing here but I guess it's a missing TEST_REQUIRE()?

On 24.09.25 17:10, Patrick Roy wrote:

From: Patrick Roy roypat@amazon.co.uk

Add AS_NO_DIRECT_MAP for mappings where direct map entries of folios are set to not present . Currently, mappings that match this description are secretmem mappings (memfd_secret()). Later, some guest_memfd configurations will also fall into this category.

Reject this new type of mappings in all locations that currently reject secretmem mappings, on the assumption that if secretmem mappings are rejected somewhere, it is precisely because of an inability to deal with folios without direct map entries, and then make memfd_secret() use AS_NO_DIRECT_MAP on its address_space to drop its special vma_is_secretmem()/secretmem_mapping() checks.

This drops a optimization in gup_fast_folio_allowed() where secretmem_mapping() was only called if CONFIG_SECRETMEM=y. secretmem is enabled by default since commit b758fe6df50d ("mm/secretmem: make it on by default"), so the secretmem check did not actually end up elided in most cases anymore anyway.

Use a new flag instead of overloading AS_INACCESSIBLE (which is already set by guest_memfd) because not all guest_memfd mappings will end up being direct map removed (e.g. in pKVM setups, parts of guest_memfd that can be mapped to userspace should also be GUP-able, and generally not have restrictions on who can access it).

Acked-by: Mike Rapoport (Microsoft) rppt@kernel.org Signed-off-by: Patrick Roy roypat@amazon.co.uk

---

I enjoy seeing secretmem special-casing in common code go away.

[...]

/* @@ -2763,18 +2761,10 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags) reject_file_backed = true; /* We hold a folio reference, so we can safely access folio fields. */

• /* secretmem folios are always order-0 folios. */
• if (IS_ENABLED(CONFIG_SECRETMEM) && !folio_test_large(folio))

• check_secretmem = true;
  

• if (!reject_file_backed && !check_secretmem)

• return true;
  

Losing that optimization is not too bad I guess.

Acked-by: David Hildenbrand david@redhat.com