From: Patrick Roy patrick.roy@linux.dev

Use the per-module export functionality to allow KVM (and only KVM) access to set_direct_map_valid_noflush(). This allows guest_memfd to remove its memory from the direct map, even if KVM is built as a module.

Only do this on x86, as only x86 and arm64 support guest_memfd, and arm64 does not support building KVM as a module.

Direct map removal gives guest_memfd the same protection that memfd_secret enjoys, such as hardening against Spectre-like attacks through in-kernel gadgets.

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- arch/x86/mm/pat/set_memory.c | 1 + 1 file changed, 1 insertion(+)

diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 970981893c9b..a7a88b598d99 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -2655,6 +2655,7 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)

return __set_pages_np(page, nr); } +EXPORT_SYMBOL_FOR_MODULES(set_direct_map_valid_noflush, "kvm");

#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable)

From: Patrick Roy patrick.roy@linux.dev

After direct map removal, a TLB flush must be done to ensure that the just-unmapped memory cannot be accessed through stale TLB entries.

This export is only needed on x86, as arm64 (the only other architecture supporting guest_memfd currently) does not allow building KVM as a module.

Cc: Thomas Gleixner tglx@linutronix.de Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- arch/x86/include/asm/tlbflush.h | 3 ++- arch/x86/mm/tlb.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 00daedfefc1b..6f57f7eb621b 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -317,7 +317,6 @@ extern void flush_tlb_all(void); extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); -extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);

static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { @@ -483,6 +482,8 @@ static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) #endif #endif /* !MODULE */

+extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); + static inline void __native_tlb_flush_global(unsigned long cr4) { native_write_cr4(cr4 ^ X86_CR4_PGE); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5d221709353e..cce591d26e4c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1561,6 +1561,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)

put_flush_tlb_info(); } +EXPORT_SYMBOL_FOR_MODULES(flush_tlb_kernel_range, "kvm");

/* * This can be used from process context to figure out what the value of

From: Patrick Roy patrick.roy@linux.dev

Add GUEST_MEMFD_FLAG_NO_DIRECT_MAP flag for KVM_CREATE_GUEST_MEMFD() ioctl. When set, guest_memfd folios will be removed from the direct map after preparation, with direct map entries only restored when the folios are freed.

To ensure these folios do not end up in places where the kernel cannot deal with them, set AS_NO_DIRECT_MAP on the guest_memfd's struct address_space if GUEST_MEMFD_FLAG_NO_DIRECT_MAP is requested.

Note that this flag causes removal of direct map entries for all guest_memfd folios independent of whether they are "shared" or "private" (although current guest_memfd only supports either all folios in the "shared" state, or all folios in the "private" state if GUEST_MEMFD_FLAG_MMAP is not set). The usecase for removing direct map entries of also the shared parts of guest_memfd are a special type of non-CoCo VM where, host userspace is trusted to have access to all of guest memory, but where Spectre-style transient execution attacks through the host kernel's direct map should still be mitigated. In this setup, KVM retains access to guest memory via userspace mappings of guest_memfd, which are reflected back into KVM's memslots via userspace_addr. This is needed for things like MMIO emulation on x86_64 to work.

Direct map entries are zapped right before guest or userspace mappings of gmem folios are set up, e.g. in kvm_gmem_fault_user_mapping() or kvm_gmem_get_pfn() [called from the KVM MMU code]. The only place where a gmem folio can be allocated without being mapped anywhere is kvm_gmem_populate(), where handling potential failures of direct map removal is not possible (by the time direct map removal is attempted, the folio is already marked as prepared, meaning attempting to re-try kvm_gmem_populate() would just result in -EEXIST without fixing up the direct map state). These folios are then removed form the direct map upon kvm_gmem_get_pfn(), e.g. when they are mapped into the guest later.

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- Documentation/virt/kvm/api.rst | 22 ++++++++----- include/linux/kvm_host.h | 12 +++++++ include/uapi/linux/kvm.h | 1 + virt/kvm/guest_memfd.c | 60 ++++++++++++++++++++++++++++++++++ 4 files changed, 86 insertions(+), 9 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 01a3abef8abb..c5f54f1370c8 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6440,15 +6440,19 @@ a single guest_memfd file, but the bound ranges must not overlap). The capability KVM_CAP_GUEST_MEMFD_FLAGS enumerates the `flags` that can be specified via KVM_CREATE_GUEST_MEMFD. Currently defined flags:

- ============================ ================================================ - GUEST_MEMFD_FLAG_MMAP Enable using mmap() on the guest_memfd file - descriptor. - GUEST_MEMFD_FLAG_INIT_SHARED Make all memory in the file shared during - KVM_CREATE_GUEST_MEMFD (memory files created - without INIT_SHARED will be marked private). - Shared memory can be faulted into host userspace - page tables. Private memory cannot. - ============================ ================================================ + ============================== ================================================ + GUEST_MEMFD_FLAG_MMAP Enable using mmap() on the guest_memfd file + descriptor. + GUEST_MEMFD_FLAG_INIT_SHARED Make all memory in the file shared during + KVM_CREATE_GUEST_MEMFD (memory files created + without INIT_SHARED will be marked private). + Shared memory can be faulted into host userspace + page tables. Private memory cannot. + GUEST_MEMFD_FLAG_NO_DIRECT_MAP The guest_memfd instance will behave similarly + to memfd_secret, and unmaps the memory backing + it from the kernel's address space before + being passed off to userspace or the guest. + ============================== ================================================

When the KVM MMU performs a PFN lookup to service a guest fault and the backing guest_memfd has the GUEST_MEMFD_FLAG_MMAP set, then the fault will always be diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 27796a09d29b..d4d5306075bf 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -738,10 +738,22 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm) if (!kvm || kvm_arch_supports_gmem_init_shared(kvm)) flags |= GUEST_MEMFD_FLAG_INIT_SHARED;

+ if (kvm_arch_gmem_supports_no_direct_map()) + flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP; + return flags; } #endif

+#ifdef CONFIG_KVM_GUEST_MEMFD +#ifndef kvm_arch_gmem_supports_no_direct_map +static inline bool kvm_arch_gmem_supports_no_direct_map(void) +{ + return false; +} +#endif +#endif /* CONFIG_KVM_GUEST_MEMFD */ + #ifndef kvm_arch_has_readonly_mem static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm) { diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index dddb781b0507..60341e1ba1be 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1612,6 +1612,7 @@ struct kvm_memory_attributes { #define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd) #define GUEST_MEMFD_FLAG_MMAP (1ULL << 0) #define GUEST_MEMFD_FLAG_INIT_SHARED (1ULL << 1) +#define GUEST_MEMFD_FLAG_NO_DIRECT_MAP (1ULL << 2)

struct kvm_create_guest_memfd { __u64 size; diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 92e7f8c1f303..ec4966a47d5e 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -7,6 +7,9 @@ #include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/set_memory.h> + +#include <asm/tlbflush.h>

#include "kvm_mm.h"

@@ -76,6 +79,49 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo return 0; }

+#define KVM_GMEM_FOLIO_NO_DIRECT_MAP BIT(0) + +static bool kvm_gmem_folio_no_direct_map(struct folio *folio) +{ + return ((u64) folio->private) & KVM_GMEM_FOLIO_NO_DIRECT_MAP; +} + +static int kvm_gmem_folio_zap_direct_map(struct folio *folio) +{ + int r = 0; + unsigned long addr = (unsigned long) folio_address(folio); + u64 gmem_flags = GMEM_I(folio_inode(folio))->flags; + + if (kvm_gmem_folio_no_direct_map(folio) || !(gmem_flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP)) + goto out; + + r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio), + false); + + if (r) + goto out; + + folio->private = (void *) KVM_GMEM_FOLIO_NO_DIRECT_MAP; + flush_tlb_kernel_range(addr, addr + folio_size(folio)); + +out: + return r; +} + +static void kvm_gmem_folio_restore_direct_map(struct folio *folio) +{ + /* + * Direct map restoration cannot fail, as the only error condition + * for direct map manipulation is failure to allocate page tables + * when splitting huge pages, but this split would have already + * happened in set_direct_map_invalid_noflush() in kvm_gmem_folio_zap_direct_map(). + * Thus set_direct_map_valid_noflush() here only updates prot bits. + */ + if (kvm_gmem_folio_no_direct_map(folio)) + set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio), + true); +} + static inline void kvm_gmem_mark_prepared(struct folio *folio) { folio_mark_uptodate(folio); @@ -398,6 +444,7 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct folio *folio; vm_fault_t ret = VM_FAULT_LOCKED; + int err;

if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; @@ -423,6 +470,12 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) kvm_gmem_mark_prepared(folio); }

+ err = kvm_gmem_folio_zap_direct_map(folio); + if (err) { + ret = vmf_error(err); + goto out_folio; + } + vmf->page = folio_file_page(folio, vmf->pgoff);

out_folio: @@ -533,6 +586,8 @@ static void kvm_gmem_free_folio(struct folio *folio) kvm_pfn_t pfn = page_to_pfn(page); int order = folio_order(folio);

+ kvm_gmem_folio_restore_direct_map(folio); + kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order)); }

@@ -596,6 +651,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) /* Unmovable mappings are supposed to be marked unevictable as well. */ WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));

+ if (flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP) + mapping_set_no_direct_map(inode->i_mapping); + GMEM_I(inode)->flags = flags;

file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops); @@ -807,6 +865,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!is_prepared) r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

+ kvm_gmem_folio_zap_direct_map(folio); + folio_unlock(folio);

if (!r)

From: Patrick Roy patrick.roy@linux.dev

x86 supports GUEST_MEMFD_FLAG_NO_DIRECT_MAP whenever direct map modifications are possible (which is always the case).

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- arch/x86/include/asm/kvm_host.h | 9 +++++++++ 1 file changed, 9 insertions(+)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5a3bfa293e8b..68bd29a52f24 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -28,6 +28,7 @@ #include <linux/sched/vhost_task.h> #include <linux/call_once.h> #include <linux/atomic.h> +#include <linux/set_memory.h>

#include <asm/apic.h> #include <asm/pvclock-abi.h> @@ -2481,4 +2482,12 @@ static inline bool kvm_arch_has_irq_bypass(void) return enable_device_posted_irqs; }

+#ifdef CONFIG_KVM_GUEST_MEMFD +static inline bool kvm_arch_gmem_supports_no_direct_map(void) +{ + return can_set_direct_map(); +} +#define kvm_arch_gmem_supports_no_direct_map kvm_arch_gmem_supports_no_direct_map +#endif /* CONFIG_KVM_GUEST_MEMFD */ + #endif /* _ASM_X86_KVM_HOST_H */

From: Patrick Roy patrick.roy@linux.dev

If guest memory is backed using a VMA that does not allow GUP (e.g. a userspace mapping of guest_memfd when the fd was allocated using GUEST_MEMFD_FLAG_NO_DIRECT_MAP), then directly loading the test ELF binary into it via read(2) potentially does not work. To nevertheless support loading binaries in this cases, do the read(2) syscall using a bounce buffer, and then memcpy from the bounce buffer into guest memory.

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- .../testing/selftests/kvm/include/test_util.h | 1 + tools/testing/selftests/kvm/lib/elf.c | 8 +++---- tools/testing/selftests/kvm/lib/io.c | 23 +++++++++++++++++++ 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index b4872ba8ed12..8140e59b59e5 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -48,6 +48,7 @@ do { \

ssize_t test_write(int fd, const void *buf, size_t count); ssize_t test_read(int fd, void *buf, size_t count); +ssize_t test_read_bounce(int fd, void *buf, size_t count); int test_seq_read(const char *path, char **bufp, size_t *sizep);

void __printf(5, 6) test_assert(bool exp, const char *exp_str, diff --git a/tools/testing/selftests/kvm/lib/elf.c b/tools/testing/selftests/kvm/lib/elf.c index f34d926d9735..e829fbe0a11e 100644 --- a/tools/testing/selftests/kvm/lib/elf.c +++ b/tools/testing/selftests/kvm/lib/elf.c @@ -31,7 +31,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) * the real size of the ELF header. */ unsigned char ident[EI_NIDENT]; - test_read(fd, ident, sizeof(ident)); + test_read_bounce(fd, ident, sizeof(ident)); TEST_ASSERT((ident[EI_MAG0] == ELFMAG0) && (ident[EI_MAG1] == ELFMAG1) && (ident[EI_MAG2] == ELFMAG2) && (ident[EI_MAG3] == ELFMAG3), "ELF MAGIC Mismatch,\n" @@ -79,7 +79,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) offset_rv = lseek(fd, 0, SEEK_SET); TEST_ASSERT(offset_rv == 0, "Seek to ELF header failed,\n" " rv: %zi expected: %i", offset_rv, 0); - test_read(fd, hdrp, sizeof(*hdrp)); + test_read_bounce(fd, hdrp, sizeof(*hdrp)); TEST_ASSERT(hdrp->e_phentsize == sizeof(Elf64_Phdr), "Unexpected physical header size,\n" " hdrp->e_phentsize: %x\n" @@ -146,7 +146,7 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename)

/* Read in the program header. */ Elf64_Phdr phdr; - test_read(fd, &phdr, sizeof(phdr)); + test_read_bounce(fd, &phdr, sizeof(phdr));

/* Skip if this header doesn't describe a loadable segment. */ if (phdr.p_type != PT_LOAD) @@ -187,7 +187,7 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename) " expected: 0x%jx", n1, errno, (intmax_t) offset_rv, (intmax_t) phdr.p_offset); - test_read(fd, addr_gva2hva(vm, phdr.p_vaddr), + test_read_bounce(fd, addr_gva2hva(vm, phdr.p_vaddr), phdr.p_filesz); } } diff --git a/tools/testing/selftests/kvm/lib/io.c b/tools/testing/selftests/kvm/lib/io.c index fedb2a741f0b..74419becc8bc 100644 --- a/tools/testing/selftests/kvm/lib/io.c +++ b/tools/testing/selftests/kvm/lib/io.c @@ -155,3 +155,26 @@ ssize_t test_read(int fd, void *buf, size_t count)

return num_read; } + +/* Test read via intermediary buffer + * + * Same as test_read, except read(2)s happen into a bounce buffer that is memcpy'd + * to buf. For use with buffers that cannot be GUP'd (e.g. guest_memfd VMAs if + * guest_memfd was created with GUEST_MEMFD_FLAG_NO_DIRECT_MAP). + */ +ssize_t test_read_bounce(int fd, void *buf, size_t count) +{ + void *bounce_buffer; + ssize_t num_read; + + TEST_ASSERT(count >= 0, "Unexpected count, count: %li", count); + + bounce_buffer = malloc(count); + TEST_ASSERT(bounce_buffer != NULL, "Failed to allocate bounce buffer"); + + num_read = test_read(fd, bounce_buffer, count); + memcpy(buf, bounce_buffer, num_read); + free(bounce_buffer); + + return num_read; +}

From: Patrick Roy patrick.roy@linux.dev

Allow selftests to configure their memslots such that userspace_addr is set to a MAP_SHARED mapping of the guest_memfd that's associated with the memslot. This setup is the configuration for non-CoCo VMs, where all guest memory is backed by a guest_memfd whose folios are all marked shared, but KVM is still able to access guest memory to provide functionality such as MMIO emulation on x86.

Add backing types for normal guest_memfd, as well as direct map removed guest_memfd.

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- .../testing/selftests/kvm/include/kvm_util.h | 18 ++++++ .../testing/selftests/kvm/include/test_util.h | 7 +++ tools/testing/selftests/kvm/lib/kvm_util.c | 61 ++++++++++--------- tools/testing/selftests/kvm/lib/test_util.c | 8 +++ 4 files changed, 65 insertions(+), 29 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 81f4355ff28a..6689b43810c1 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -641,6 +641,24 @@ static inline bool is_smt_on(void)

void vm_create_irqchip(struct kvm_vm *vm);

+static inline uint32_t backing_src_guest_memfd_flags(enum vm_mem_backing_src_type t) +{ + uint32_t flags = 0; + + switch (t) { + case VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP: + flags |= GUEST_MEMFD_FLAG_NO_DIRECT_MAP; + fallthrough; + case VM_MEM_SRC_GUEST_MEMFD: + flags |= GUEST_MEMFD_FLAG_MMAP | GUEST_MEMFD_FLAG_INIT_SHARED; + break; + default: + break; + } + + return flags; +} + static inline int __vm_create_guest_memfd(struct kvm_vm *vm, uint64_t size, uint64_t flags) { diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index 8140e59b59e5..ea6de20ce8ef 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -152,6 +152,8 @@ enum vm_mem_backing_src_type { VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB, VM_MEM_SRC_SHMEM, VM_MEM_SRC_SHARED_HUGETLB, + VM_MEM_SRC_GUEST_MEMFD, + VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP, NUM_SRC_TYPES, };

@@ -184,6 +186,11 @@ static inline bool backing_src_is_shared(enum vm_mem_backing_src_type t) return vm_mem_backing_src_alias(t)->flag & MAP_SHARED; }

+static inline bool backing_src_is_guest_memfd(enum vm_mem_backing_src_type t) +{ + return t == VM_MEM_SRC_GUEST_MEMFD || t == VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP; +} + static inline bool backing_src_can_be_huge(enum vm_mem_backing_src_type t) { return t != VM_MEM_SRC_ANONYMOUS && t != VM_MEM_SRC_SHMEM; diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 56ddbca91850..28ee51253909 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1013,6 +1013,33 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, alignment = 1; #endif

+ if (guest_memfd < 0) { + if ((flags & KVM_MEM_GUEST_MEMFD) || backing_src_is_guest_memfd(src_type)) { + uint32_t guest_memfd_flags = backing_src_guest_memfd_flags(src_type); + + TEST_ASSERT(!guest_memfd_offset, + "Offset must be zero when creating new guest_memfd"); + guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); + } + } else { + /* + * Install a unique fd for each memslot so that the fd + * can be closed when the region is deleted without + * needing to track if the fd is owned by the framework + * or by the caller. + */ + guest_memfd = kvm_dup(guest_memfd); + } + + if (guest_memfd > 0) { + flags |= KVM_MEM_GUEST_MEMFD; + + region->region.guest_memfd = guest_memfd; + region->region.guest_memfd_offset = guest_memfd_offset; + } else { + region->region.guest_memfd = -1; + } + /* * When using THP mmap is not guaranteed to returned a hugepage aligned * address so we have to pad the mmap. Padding is not needed for HugeTLB @@ -1028,10 +1055,13 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, if (alignment > 1) region->mmap_size += alignment;

- region->fd = -1; - if (backing_src_is_shared(src_type)) + if (backing_src_is_guest_memfd(src_type)) + region->fd = guest_memfd; + else if (backing_src_is_shared(src_type)) region->fd = kvm_memfd_alloc(region->mmap_size, src_type == VM_MEM_SRC_SHARED_HUGETLB); + else + region->fd = -1;

region->mmap_start = kvm_mmap(region->mmap_size, PROT_READ | PROT_WRITE, vm_mem_backing_src_alias(src_type)->flag, @@ -1056,33 +1086,6 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, }

region->backing_src_type = src_type; - - if (guest_memfd < 0) { - if (flags & KVM_MEM_GUEST_MEMFD) { - uint32_t guest_memfd_flags = 0; - TEST_ASSERT(!guest_memfd_offset, - "Offset must be zero when creating new guest_memfd"); - guest_memfd = vm_create_guest_memfd(vm, mem_size, guest_memfd_flags); - } - } else { - /* - * Install a unique fd for each memslot so that the fd - * can be closed when the region is deleted without - * needing to track if the fd is owned by the framework - * or by the caller. - */ - guest_memfd = kvm_dup(guest_memfd); - } - - if (guest_memfd > 0) { - flags |= KVM_MEM_GUEST_MEMFD; - - region->region.guest_memfd = guest_memfd; - region->region.guest_memfd_offset = guest_memfd_offset; - } else { - region->region.guest_memfd = -1; - } - region->unused_phy_pages = sparsebit_alloc(); if (vm_arch_has_protected_memory(vm)) region->protected_phy_pages = sparsebit_alloc(); diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index 8a1848586a85..ce9fe0271515 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -306,6 +306,14 @@ const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i) */ .flag = MAP_SHARED, }, + [VM_MEM_SRC_GUEST_MEMFD] = { + .name = "guest_memfd", + .flag = MAP_SHARED, + }, + [VM_MEM_SRC_GUEST_MEMFD_NO_DIRECT_MAP] = { + .name = "guest_memfd_no_direct_map", + .flag = MAP_SHARED, + } }; _Static_assert(ARRAY_SIZE(aliases) == NUM_SRC_TYPES, "Missing new backing src types?");

From: Patrick Roy patrick.roy@linux.dev

Use one of the padding fields in struct vm_shape to carry an enum vm_mem_backing_src_type value, to give the option to overwrite the default of VM_MEM_SRC_ANONYMOUS in __vm_create().

Overwriting this default will allow tests to create VMs where the test code is backed by mmap'd guest_memfd instead of anonymous memory.

Signed-off-by: Patrick Roy patrick.roy@linux.dev Signed-off-by: Nikita Kalyazin kalyazin@amazon.com --- .../testing/selftests/kvm/include/kvm_util.h | 19 ++++++++++--------- tools/testing/selftests/kvm/lib/kvm_util.c | 2 +- tools/testing/selftests/kvm/lib/x86/sev.c | 1 + .../selftests/kvm/pre_fault_memory_test.c | 1 + 4 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 6689b43810c1..4bc4af9a40cf 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -192,7 +192,7 @@ enum vm_guest_mode { struct vm_shape { uint32_t type; uint8_t mode; - uint8_t pad0; + uint8_t src_type; uint16_t pad1; };

@@ -200,14 +200,15 @@ kvm_static_assert(sizeof(struct vm_shape) == sizeof(uint64_t));

#define VM_TYPE_DEFAULT 0

-#define VM_SHAPE(__mode) \ -({ \ - struct vm_shape shape = { \ - .mode = (__mode), \ - .type = VM_TYPE_DEFAULT \ - }; \ - \ - shape; \ +#define VM_SHAPE(__mode) \ +({ \ + struct vm_shape shape = { \ + .mode = (__mode), \ + .type = VM_TYPE_DEFAULT, \ + .src_type = VM_MEM_SRC_ANONYMOUS \ + }; \ + \ + shape; \ })

#if defined(__aarch64__) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 28ee51253909..268a4520633b 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -467,7 +467,7 @@ struct kvm_vm *__vm_create(struct vm_shape shape, uint32_t nr_runnable_vcpus, if (is_guest_memfd_required(shape)) flags |= KVM_MEM_GUEST_MEMFD;

- vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, 0, 0, nr_pages, flags); + vm_userspace_mem_region_add(vm, shape.src_type, 0, 0, nr_pages, flags); for (i = 0; i < NR_MEM_REGIONS; i++) vm->memslots[i] = 0;

diff --git a/tools/testing/selftests/kvm/lib/x86/sev.c b/tools/testing/selftests/kvm/lib/x86/sev.c index c3a9838f4806..d920880e4fc0 100644 --- a/tools/testing/selftests/kvm/lib/x86/sev.c +++ b/tools/testing/selftests/kvm/lib/x86/sev.c @@ -164,6 +164,7 @@ struct kvm_vm *vm_sev_create_with_one_vcpu(uint32_t type, void *guest_code, struct vm_shape shape = { .mode = VM_MODE_DEFAULT, .type = type, + .src_type = VM_MEM_SRC_ANONYMOUS, }; struct kvm_vm *vm; struct kvm_vcpu *cpus[1]; diff --git a/tools/testing/selftests/kvm/pre_fault_memory_test.c b/tools/testing/selftests/kvm/pre_fault_memory_test.c index 93e603d91311..8a4d5af53fab 100644 --- a/tools/testing/selftests/kvm/pre_fault_memory_test.c +++ b/tools/testing/selftests/kvm/pre_fault_memory_test.c @@ -165,6 +165,7 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) const struct vm_shape shape = { .mode = VM_MODE_DEFAULT, .type = vm_type, + .src_type = VM_MEM_SRC_ANONYMOUS, }; struct kvm_vcpu *vcpu; struct kvm_run *run;

On 12/5/25 17:58, Kalyazin, Nikita wrote:

+static int kvm_gmem_folio_zap_direct_map(struct folio *folio) +{

• int r = 0;
• unsigned long addr = (unsigned long) folio_address(folio);
• u64 gmem_flags = GMEM_I(folio_inode(folio))->flags;
• if (kvm_gmem_folio_no_direct_map(folio) || !(gmem_flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP))

• goto out;
  

• r = set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),

• 			 false);
  

• if (r)

• goto out;
  

• folio->private = (void *) KVM_GMEM_FOLIO_NO_DIRECT_MAP;

With Dave's suggestion on patch 1/13 to have folio_zap_direct_map(), setting this folio->private flag wouldn't be possible between the zap and tlb flush, but it's not an issue to set it before the zap, right?

• flush_tlb_kernel_range(addr, addr + folio_size(folio));

+out:

• return r;

+}

+static void kvm_gmem_folio_restore_direct_map(struct folio *folio) +{

• /*

• * Direct map restoration cannot fail, as the only error condition
  

• * for direct map manipulation is failure to allocate page tables
  

• * when splitting huge pages, but this split would have already
  

• * happened in set_direct_map_invalid_noflush() in kvm_gmem_folio_zap_direct_map().
  

• * Thus set_direct_map_valid_noflush() here only updates prot bits.
  

• */
  

• if (kvm_gmem_folio_no_direct_map(folio))

• set_direct_map_valid_noflush(folio_page(folio, 0), folio_nr_pages(folio),
  

• 			 true);
  

I think you're missing here clearing KVM_GMEM_FOLIO_NO_DIRECT_MAP from folio->private, which means if there's another kvm_gmem_folio_zap_direct_map() call on it in the future, it will do nothing?

+}

static inline void kvm_gmem_mark_prepared(struct folio *folio) { folio_mark_uptodate(folio); @@ -398,6 +444,7 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); struct folio *folio; vm_fault_t ret = VM_FAULT_LOCKED;

• int err;

if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; @@ -423,6 +470,12 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) kvm_gmem_mark_prepared(folio); }

• err = kvm_gmem_folio_zap_direct_map(folio);
• if (err) {

• ret = vmf_error(err);
  

• goto out_folio;
  

• }
• vmf->page = folio_file_page(folio, vmf->pgoff);

out_folio: @@ -533,6 +586,8 @@ static void kvm_gmem_free_folio(struct folio *folio) kvm_pfn_t pfn = page_to_pfn(page); int order = folio_order(folio);

• kvm_gmem_folio_restore_direct_map(folio);
• kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));

} @@ -596,6 +651,9 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) /* Unmovable mappings are supposed to be marked unevictable as well. */ WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));

• if (flags & GUEST_MEMFD_FLAG_NO_DIRECT_MAP)

• mapping_set_no_direct_map(inode->i_mapping);
  

• GMEM_I(inode)->flags = flags;

file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops); @@ -807,6 +865,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, if (!is_prepared) r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);

• kvm_gmem_folio_zap_direct_map(folio);
• folio_unlock(folio);

if (!r)