On 20.06.24 18:36, Jason Gunthorpe wrote:
On Thu, Jun 20, 2024 at 04:45:08PM +0200, David Hildenbrand wrote:
If we could disallow pinning any shared pages, that would make life a lot easier, but I think there were reasons for why we might require it. To convert shared->private, simply unmap that folio (only the shared parts could possibly be mapped) from all user page tables.
IMHO it should be reasonable to make it work like ZONE_MOVABLE and FOLL_LONGTERM. Making a shared page private is really no different from moving it.
And if you have built a VMM that uses VMA mapped shared pages and short-term pinning then you should really also ensure that the VM is aware when the pins go away. For instance if you are doing some virtio thing with O_DIRECT pinning then the guest will know the pins are gone when it observes virtio completions.
In this way making private is just like moving, we unmap the page and then drive the refcount to zero, then move it.
Yes, but here is the catch: what if a single shared subpage of a large folio is (validly) longterm pinned and you want to convert another shared subpage to private?
Sure, we can unmap the whole large folio (including all shared parts) before the conversion, just like we would do for migration. But we cannot detect that nobody pinned that subpage that we want to convert to private.
Core-mm is not, and will not, track pins per subpage.
So I only see two options:
a) Disallow long-term pinning. That means, we can, with a bit of wait, always convert subpages shared->private after unmapping them and waiting for the short-term pin to go away. Not too bad, and we already have other mechanisms disallow long-term pinnings (especially writable fs ones!).
b) Expose the large folio as multiple 4k folios to the core-mm.
b) would look as follows: we allocate a gigantic page from the (hugetlb) reserve into guest_memfd. Then, we break it down into individual 4k folios by splitting/demoting the folio. We make sure that all 4k folios are unmovable (raised refcount). We keep tracking internally that these 4k folios comprise a single large gigantic page.
Core-mm can track for us now without any modifications per (previously subpage,) now small folios GUP pins and page table mappings without modifications.
Once we unmap the gigantic page from guest_memfd, we recronstruct the gigantic page and hand it back to the reserve (only possible once all pins are gone).
We can still map the whole thing into the KVM guest+iommu using a single large unit, because guest_memfd knows the origin/relationship of these pages. But we would only map individual pages into user page tables (unless we use large VM_PFNMAP mappings, but then also pinning would not work, so that's likely also not what we want).
The downside is that we won't benefit from vmemmap optimizations for large folios from hugetlb, and have more tracking overhead when mapping individual pages into user page tables.
OTOH, maybe we really *need* per-page tracking and this might be the simplest way forward, making GUP and friends just work naturally with it.
I'm kind of surprised the CC folks don't want the same thing for exactly the same reason. It is much easier to recover the huge mappings for the S2 in the presence of shared holes if you track it this way. Even CC will have this problem, to some degree, too.
Precisely! RH (and therefore, me) is primarily interested in existing guest_memfd users at this point ("CC"), and I don't see an easy way to get that running with huge pages in the existing model reasonably well ...
IMHO it is an important topic so I'm glad you are thinking about it.
Thank my manager ;)
There is definately some overlap here where if you do teach guest_memfd about huge pages then you must also provide a away to map the fragments of them that have become shared. I think there is little option here unless you double allocate and/or destroy the performance properties of the huge pages.
Right, and that's not what we want.
It is just the nature of our system that shared pages must be in VMAs and must be copy_to/from_user/GUP'able/etc.
Right. Longterm GUP is not a real requirement.