Re: [PATCH v3 09/23] mm/gup: introduce pin_user_pages*() and FOLL_PIN

On Mon, Nov 11, 2019 at 04:06:46PM -0800, John Hubbard wrote:

Introduce pin_user_pages*() variations of get_user_pages*() calls, and also pin_longterm_pages*() variations.

These variants all set FOLL_PIN, which is also introduced, and thoroughly documented.

The pin_longterm*() variants also set FOLL_LONGTERM, in addition to FOLL_PIN:

pin_user_pages()
pin_user_pages_remote()
pin_user_pages_fast()

pin_longterm_pages()
pin_longterm_pages_remote()
pin_longterm_pages_fast()

All pages that are pinned via the above calls, must be unpinned via put_user_page().

The underlying rules are:

• These are gup-internal flags, so the call sites should not directly

set FOLL_PIN nor FOLL_LONGTERM. That behavior is enforced with assertions, for the new FOLL_PIN flag. However, for the pre-existing FOLL_LONGTERM flag, which has some call sites that still directly set FOLL_LONGTERM, there is no assertion yet.

• Call sites that want to indicate that they are going to do DirectIO ("DIO") or something with similar characteristics, should call a get_user_pages()-like wrapper call that sets FOLL_PIN. These wrappers will: * Start with "pin_user_pages" instead of "get_user_pages". That makes it easy to find and audit the call sites. * Set FOLL_PIN

• For pages that are received via FOLL_PIN, those pages must be returned via put_user_page().

Thanks to Jan Kara and Vlastimil Babka for explaining the 4 cases in this documentation. (I've reworded it and expanded upon it.)

Reviewed-by: Jérôme Glisse jglisse@redhat.com Cc: Mike Rapoport rppt@kernel.org Cc: Jonathan Corbet corbet@lwn.net Cc: Ira Weiny ira.weiny@intel.com Signed-off-by: John Hubbard jhubbard@nvidia.com

---

Reviewed-by: Mike Rapoport rppt@linux.ibm.com # Documentation

Documentation/core-api/index.rst | 1 + Documentation/core-api/pin_user_pages.rst | 218 ++++++++++++++++++ include/linux/mm.h | 62 +++++- mm/gup.c | 260 ++++++++++++++++++++-- 4 files changed, 514 insertions(+), 27 deletions(-) create mode 100644 Documentation/core-api/pin_user_pages.rst

diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst index ab0eae1c153a..413f7d7c8642 100644 --- a/Documentation/core-api/index.rst +++ b/Documentation/core-api/index.rst @@ -31,6 +31,7 @@ Core utilities generic-radix-tree memory-allocation mm-api

• pin_user_pages gfp_mask-from-fs-io timekeeping boot-time-mm

diff --git a/Documentation/core-api/pin_user_pages.rst b/Documentation/core-api/pin_user_pages.rst new file mode 100644 index 000000000000..ce819e709435 --- /dev/null +++ b/Documentation/core-api/pin_user_pages.rst @@ -0,0 +1,218 @@ +.. SPDX-License-Identifier: GPL-2.0

+==================================================== +pin_user_pages() and related calls +====================================================

+.. contents:: :local:

+Overview +========

+This document describes the following functions: ::

• pin_user_pages
• pin_user_pages_fast
• pin_user_pages_remote
• pin_longterm_pages
• pin_longterm_pages_fast
• pin_longterm_pages_remote

+Basic description of FOLL_PIN +=============================

+FOLL_PIN and FOLL_LONGTERM are flags that can be passed to the get_user_pages*() +("gup") family of functions. FOLL_PIN has significant interactions and +interdependencies with FOLL_LONGTERM, so both are covered here.

+Both FOLL_PIN and FOLL_LONGTERM are internal to gup, meaning that neither +FOLL_PIN nor FOLL_LONGTERM should not appear at the gup call sites. This allows +the associated wrapper functions (pin_user_pages() and others) to set the +correct combination of these flags, and to check for problems as well.

+FOLL_PIN and FOLL_GET are mutually exclusive for a given gup call. However, +multiple threads and call sites are free to pin the same struct pages, via both +FOLL_PIN and FOLL_GET. It's just the call site that needs to choose one or the +other, not the struct page(s).

+The FOLL_PIN implementation is nearly the same as FOLL_GET, except that FOLL_PIN +uses a different reference counting technique.

+FOLL_PIN is a prerequisite to FOLL_LONGTGERM. Another way of saying that is, +FOLL_LONGTERM is a specific case, more restrictive case of FOLL_PIN.

+Which flags are set by each wrapper +===================================

+Only FOLL_PIN and FOLL_LONGTERM are covered here. These flags are added to +whatever flags the caller provides::

• Function gup flags (FOLL_PIN or FOLL_LONGTERM only)

• ---

• pin_user_pages FOLL_PIN
• pin_user_pages_fast FOLL_PIN
• pin_user_pages_remote FOLL_PIN
• pin_longterm_pages FOLL_PIN | FOLL_LONGTERM
• pin_longterm_pages_fast FOLL_PIN | FOLL_LONGTERM
• pin_longterm_pages_remote FOLL_PIN | FOLL_LONGTERM

+Tracking dma-pinned pages +=========================

+Some of the key design constraints, and solutions, for tracking dma-pinned +pages:

+* An actual reference count, per struct page, is required. This is because

• multiple processes may pin and unpin a page.

+* False positives (reporting that a page is dma-pinned, when in fact it is not)

• are acceptable, but false negatives are not.

+* struct page may not be increased in size for this, and all fields are already

• used.

+* Given the above, we can overload the page->_refcount field by using, sort of,

• the upper bits in that field for a dma-pinned count. "Sort of", means that,
• rather than dividing page->_refcount into bit fields, we simple add a medium-
• large value (GUP_PIN_COUNTING_BIAS, initially chosen to be 1024: 10 bits) to
• page->_refcount. This provides fuzzy behavior: if a page has get_page() called
• on it 1024 times, then it will appear to have a single dma-pinned count.
• And again, that's acceptable.

+This also leads to limitations: there are only 31-10==21 bits available for a +counter that increments 10 bits at a time.

+TODO: for 1GB and larger huge pages, this is cutting it close. That's because +when pin_user_pages() follows such pages, it increments the head page by "1" +(where "1" used to mean "+1" for get_user_pages(), but now means "+1024" for +pin_user_pages()) for each tail page. So if you have a 1GB huge page:

+* There are 256K (18 bits) worth of 4 KB tail pages. +* There are 21 bits available to count up via GUP_PIN_COUNTING_BIAS (that is,

• 10 bits at a time)

+* There are 21 - 18 == 3 bits available to count. Except that there aren't,

• because you need to allow for a few normal get_page() calls on the head page,
• as well. Fortunately, the approach of using addition, rather than "hard"
• bitfields, within page->_refcount, allows for sharing these bits gracefully.
• But we're still looking at about 8 references.

+This, however, is a missing feature more than anything else, because it's easily +solved by addressing an obvious inefficiency in the original get_user_pages() +approach of retrieving pages: stop treating all the pages as if they were +PAGE_SIZE. Retrieve huge pages as huge pages. The callers need to be aware of +this, so some work is required. Once that's in place, this limitation mostly +disappears from view, because there will be ample refcounting range available.

+* Callers must specifically request "dma-pinned tracking of pages". In other

• words, just calling get_user_pages() will not suffice; a new set of functions,
• pin_user_page() and related, must be used.

+FOLL_PIN, FOLL_GET, FOLL_LONGTERM: when to use which flags +==========================================================

+Thanks to Jan Kara, Vlastimil Babka and several other -mm people, for describing +these categories:

+CASE 1: Direct IO (DIO) +----------------------- +There are GUP references to pages that are serving +as DIO buffers. These buffers are needed for a relatively short time (so they +are not "long term"). No special synchronization with page_mkclean() or +munmap() is provided. Therefore, flags to set at the call site are: ::

• FOLL_PIN

+...but rather than setting FOLL_PIN directly, call sites should use one of +the pin_user_pages*() routines that set FOLL_PIN.

+CASE 2: RDMA +------------ +There are GUP references to pages that are serving as DMA +buffers. These buffers are needed for a long time ("long term"). No special +synchronization with page_mkclean() or munmap() is provided. Therefore, flags +to set at the call site are: ::

• FOLL_PIN | FOLL_LONGTERM

+NOTE: Some pages, such as DAX pages, cannot be pinned with longterm pins. That's +because DAX pages do not have a separate page cache, and so "pinning" implies +locking down file system blocks, which is not (yet) supported in that way.

+CASE 3: Hardware with page faulting support +------------------------------------------- +Here, a well-written driver doesn't normally need to pin pages at all. However, +if the driver does choose to do so, it can register MMU notifiers for the range, +and will be called back upon invalidation. Either way (avoiding page pinning, or +using MMU notifiers to unpin upon request), there is proper synchronization with +both filesystem and mm (page_mkclean(), munmap(), etc).

+Therefore, neither flag needs to be set.

+In this case, ideally, neither get_user_pages() nor pin_user_pages() should be +called. Instead, the software should be written so that it does not pin pages. +This allows mm and filesystems to operate more efficiently and reliably.

+CASE 4: Pinning for struct page manipulation only +------------------------------------------------- +Here, normal GUP calls are sufficient, so neither flag needs to be set.

+page_dma_pinned(): the whole point of pinning +=============================================

+The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able +to query, "is this page DMA-pinned?" That allows code such as page_mkclean() +(and file system writeback code in general) to make informed decisions about +what to do when a page cannot be unmapped due to such pins.

+What to do in those cases is the subject of a years-long series of discussions +and debates (see the References at the end of this document). It's a TODO item +here: fill in the details once that's worked out. Meanwhile, it's safe to say +that having this available: ::

•    static inline bool page_dma_pinned(struct page *page)
  

+...is a prerequisite to solving the long-running gup+DMA problem.

+Another way of thinking about FOLL_GET, FOLL_PIN, and FOLL_LONGTERM +===================================================================

+Another way of thinking about these flags is as a progression of restrictions: +FOLL_GET is for struct page manipulation, without affecting the data that the +struct page refers to. FOLL_PIN is a *replacement* for FOLL_GET, and is for +short term pins on pages whose data *will* get accessed. As such, FOLL_PIN is +a "more severe" form of pinning. And finally, FOLL_LONGTERM is an even more +restrictive case that has FOLL_PIN as a prerequisite: this is for pages that +will be pinned longterm, and whose data will be accessed.

+Unit testing +============ +This file::

• tools/testing/selftests/vm/gup_benchmark.c

+has the following new calls to exercise the new pin*() wrapper functions:

+* PIN_FAST_BENCHMARK (./gup_benchmark -a) +* PIN_LONGTERM_BENCHMARK (./gup_benchmark -a) +* PIN_BENCHMARK (./gup_benchmark -a)

+You can monitor how many total dma-pinned pages have been acquired and released +since the system was booted, via two new /proc/vmstat entries: ::

• /proc/vmstat/nr_foll_pin_requested
• /proc/vmstat/nr_foll_pin_requested

+Those are both going to show zero, unless CONFIG_DEBUG_VM is set. This is +because there is a noticeable performance drop in put_user_page(), when they +are activated.

+References +==========

+* `Some slow progress on get_user_pages() (Apr 2, 2019) https://lwn.net/Articles/784574/`_ +* `DMA and get_user_pages() (LPC: Dec 12, 2018) https://lwn.net/Articles/774411/`_ +* `The trouble with get_user_pages() (Apr 30, 2018) https://lwn.net/Articles/753027/`_

+John Hubbard, October, 2019 diff --git a/include/linux/mm.h b/include/linux/mm.h index 96228376139c..11e0086d64a4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1542,9 +1542,23 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked); +long pin_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,

• 	   unsigned long start, unsigned long nr_pages,
  

• 	   unsigned int gup_flags, struct page **pages,
  

• 	   struct vm_area_struct **vmas, int *locked);
  

+long pin_longterm_pages_remote(struct task_struct *tsk, struct mm_struct *mm,

• 	       unsigned long start, unsigned long nr_pages,
  

• 	       unsigned int gup_flags, struct page **pages,
  

• 	       struct vm_area_struct **vmas, int *locked);
  

long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas); +long pin_user_pages(unsigned long start, unsigned long nr_pages,

•     unsigned int gup_flags, struct page **pages,
  

•     struct vm_area_struct **vmas);
  

+long pin_longterm_pages(unsigned long start, unsigned long nr_pages,

• 	unsigned int gup_flags, struct page **pages,
  

• 	struct vm_area_struct **vmas);
  

long get_user_pages_locked(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, int *locked); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, @@ -1552,6 +1566,10 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, int get_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages); +int pin_user_pages_fast(unsigned long start, int nr_pages,

• 	unsigned int gup_flags, struct page **pages);
  

+int pin_longterm_pages_fast(unsigned long start, int nr_pages,

• 	    unsigned int gup_flags, struct page **pages);
  

int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc); int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, @@ -2610,13 +2628,15 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_ANON 0x8000 /* don't do file mappings */ #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ #define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ +#define FOLL_PIN 0x40000 /* pages must be released via put_user_page() */ /*

• • NOTE on FOLL_LONGTERM:

• • FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each
• • other. Here is what they mean, and how to use them:
  • FOLL_LONGTERM indicates that the page will be held for an indefinite time

• • period _often_ under userspace control. This is contrasted with
• • iov_iter_get_pages() where usages which are transient.

• • period _often_ under userspace control. This is in contrast to
• • iov_iter_get_pages(), where usages which are transient.
  • FIXME: For pages which are part of a filesystem, mappings are subject to the
  • lifetime enforced by the filesystem and we need guarantees that longterm

@@ -2631,11 +2651,41 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,

• Currently only get_user_pages() and get_user_pages_fast() support this flag
• and calls to get_user_pages_[un]locked are specifically not allowed. This
• is due to an incompatibility with the FS DAX check and

• • FAULT_FLAG_ALLOW_RETRY

• • FAULT_FLAG_ALLOW_RETRY.

• • In the CMA case: longterm pins in a CMA region would unnecessarily fragment
• • that region. And so CMA attempts to migrate the page before pinning when

• • In the CMA case: long term pins in a CMA region would unnecessarily fragment
• • that region. And so, CMA attempts to migrate the page before pinning, when
  • FOLL_LONGTERM is specified.
• • FOLL_PIN indicates that a special kind of tracking (not just page->_refcount,
• • but an additional pin counting system) will be invoked. This is intended for
• • anything that gets a page reference and then touches page data (for example,
• • Direct IO). This lets the filesystem know that some non-file-system entity is
• • potentially changing the pages' data. In contrast to FOLL_GET (whose pages
• • are released via put_page()), FOLL_PIN pages must be released, ultimately, by
• • a call to put_user_page().
• • FOLL_PIN is similar to FOLL_GET: both of these pin pages. They use different
• • and separate refcounting mechanisms, however, and that means that each has
• • its own acquire and release mechanisms:

• • FOLL_GET: get_user_pages*() to acquire, and put_page() to release.
    

• • FOLL_PIN: pin_user_pages*() or pin_longterm_pages*() to acquire, and
    

• •           put_user_pages to release.
    

• • FOLL_PIN and FOLL_GET are mutually exclusive for a given function call.
• • (The underlying pages may experience both FOLL_GET-based and FOLL_PIN-based
• • calls applied to them, and that's perfectly OK. This is a constraint on the
• • callers, not on the pages.)
• • FOLL_PIN and FOLL_LONGTERM should be set internally by the pin_user_page*()
• • and pin_longterm_*() APIs, never directly by the caller. That's in order to
• • help avoid mismatches when releasing pages: get_user_pages*() pages must be
• • released via put_page(), while pin_user_pages*() pages must be released via
• • put_user_page().
• • Please see Documentation/vm/pin_user_pages.rst for more information.
  */

static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) diff --git a/mm/gup.c b/mm/gup.c index cfe6dc5fc343..ea31810da828 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -194,6 +194,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, spinlock_t *ptl; pte_t *ptep, pte;

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==

• 	 (FOLL_PIN | FOLL_GET)))
  

• return ERR_PTR(-EINVAL);
  

retry: if (unlikely(pmd_bad(*pmd))) return no_page_table(vma, flags); @@ -805,7 +809,7 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm, start = untagged_addr(start);

• VM_BUG_ON(!!pages != !!(gup_flags & FOLL_GET));

• VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN)));

/* * If FOLL_FORCE is set then do not force a full fault as the hinting @@ -1029,7 +1033,16 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk, BUG_ON(*locked != 1); }

• if (pages)

• /*

• * FOLL_PIN and FOLL_GET are mutually exclusive. Traditional behavior
  

• * is to set FOLL_GET if the caller wants pages[] filled in (but has
  

• * carelessly failed to specify FOLL_GET), so keep doing that, but only
  

• * for FOLL_GET, not for the newer FOLL_PIN.
  

• *
  

• * FOLL_PIN always expects pages to be non-null, but no need to assert
  

• * that here, as any failures will be obvious enough.
  

• */
  

• if (pages && !(flags & FOLL_PIN)) flags |= FOLL_GET;

pages_done = 0; @@ -1166,6 +1179,14 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) {

• /*

• * FOLL_PIN must only be set internally by the pin_user_page*() and
  

• * pin_longterm_*() APIs, never directly by the caller, so enforce that
  

• * with an assertion:
  

• */
  

• if (WARN_ON_ONCE(gup_flags & FOLL_PIN))

• return -EINVAL;
  

• /*
  • Current FOLL_LONGTERM behavior is incompatible with
  • FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on

@@ -1626,6 +1647,14 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) {

• /*

• * FOLL_PIN must only be set internally by the pin_user_page*() and
  

• * pin_longterm_*() APIs, never directly by the caller, so enforce that
  

• * with an assertion:
  

• */
  

• if (WARN_ON_ONCE(gup_flags & FOLL_PIN))

• return -EINVAL;
  

• return __gup_longterm_locked(current, current->mm, start, nr_pages, pages, vmas, gup_flags | FOLL_TOUCH);

} @@ -2377,29 +2406,14 @@ static int __gup_longterm_unlocked(unsigned long start, int nr_pages, return ret; } -/**

• • get_user_pages_fast() - pin user pages in memory
• • @start: starting user address
• • @nr_pages: number of pages from start to pin
• • @gup_flags: flags modifying pin behaviour
• • @pages: array that receives pointers to the pages pinned.

• • Should be at least nr_pages long.
    

• • Attempt to pin user pages in memory without taking mm->mmap_sem.
• • If not successful, it will fall back to taking the lock and
• • calling get_user_pages().
• • Returns number of pages pinned. This may be fewer than the number
• • requested. If nr_pages is 0 or negative, returns 0. If no pages
• • were pinned, returns -errno.
• */

-int get_user_pages_fast(unsigned long start, int nr_pages,

• 	unsigned int gup_flags, struct page **pages)
  

+static int internal_get_user_pages_fast(unsigned long start, int nr_pages,

• 			unsigned int gup_flags,
  

• 			struct page **pages)
  

{ unsigned long addr, len, end; int nr = 0, ret = 0;

• if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM)))

• if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | FOLL_PIN))) return -EINVAL;

start = untagged_addr(start) & PAGE_MASK; @@ -2439,4 +2453,208 @@ int get_user_pages_fast(unsigned long start, int nr_pages, return ret; }

+/**

• • get_user_pages_fast() - pin user pages in memory
• • @start: starting user address
• • @nr_pages: number of pages from start to pin
• • @gup_flags: flags modifying pin behaviour
• • @pages: array that receives pointers to the pages pinned.

• • Should be at least nr_pages long.
    

• • Attempt to pin user pages in memory without taking mm->mmap_sem.
• • If not successful, it will fall back to taking the lock and
• • calling get_user_pages().
• • Returns number of pages pinned. This may be fewer than the number requested.
• • If nr_pages is 0 or negative, returns 0. If no pages were pinned, returns
• • -errno.
• */

+int get_user_pages_fast(unsigned long start, int nr_pages,

• 	unsigned int gup_flags, struct page **pages)
  

+{

• /*

• * FOLL_PIN must only be set internally by the pin_user_page*() and
  

• * pin_longterm_*() APIs, never directly by the caller, so enforce that:
  

• */
  

• if (WARN_ON_ONCE(gup_flags & FOLL_PIN))

• return -EINVAL;
  

• return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages);

+} EXPORT_SYMBOL_GPL(get_user_pages_fast);

+/**

• • pin_user_pages_fast() - pin user pages in memory without taking locks
• • Nearly the same as get_user_pages_fast(), except that FOLL_PIN is set. See
• • get_user_pages_fast() for documentation on the function arguments, because
• • the arguments here are identical.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for further details.
• • This is intended for Case 1 (DIO) in Documentation/vm/pin_user_pages.rst. It
• • is NOT intended for Case 2 (RDMA: long-term pins).
• */

+int pin_user_pages_fast(unsigned long start, int nr_pages,

• 	unsigned int gup_flags, struct page **pages)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= FOLL_PIN;
• return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages);

+} +EXPORT_SYMBOL_GPL(pin_user_pages_fast);

+/**

• • pin_longterm_pages_fast() - pin user pages in memory without taking locks
• • Nearly the same as get_user_pages_fast(), except that FOLL_PIN and
• • FOLL_LONGTERM are set. See get_user_pages_fast() for documentation on the
• • function arguments, because the arguments here are identical.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for further details.
• • FOLL_LONGTERM means that the pages are being pinned for "long term" use,
• • typically by a non-CPU device, and we cannot be sure that waiting for a
• • pinned page to become unpin will be effective.
• • This is intended for Case 2 (RDMA: long-term pins) of the FOLL_PIN
• • documentation.
• */

+int pin_longterm_pages_fast(unsigned long start, int nr_pages,

• 	    unsigned int gup_flags, struct page **pages)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= (FOLL_PIN | FOLL_LONGTERM);
• return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages);

+} +EXPORT_SYMBOL_GPL(pin_longterm_pages_fast);

+/**

• • pin_user_pages_remote() - pin pages of a remote process (task != current)
• • Nearly the same as get_user_pages_remote(), except that FOLL_PIN is set. See
• • get_user_pages_remote() for documentation on the function arguments, because
• • the arguments here are identical.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for details.
• • This is intended for Case 1 (DIO) in Documentation/vm/pin_user_pages.rst. It
• • is NOT intended for Case 2 (RDMA: long-term pins).
• */

+long pin_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,

• 	   unsigned long start, unsigned long nr_pages,
  

• 	   unsigned int gup_flags, struct page **pages,
  

• 	   struct vm_area_struct **vmas, int *locked)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= FOLL_TOUCH | FOLL_REMOTE | FOLL_PIN;
• return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas,

• 		       locked, gup_flags);
  

+} +EXPORT_SYMBOL(pin_user_pages_remote);

+/**

• • pin_longterm_pages_remote() - pin pages of a remote process (task != current)
• • Nearly the same as get_user_pages_remote(), but note that FOLL_TOUCH is not
• • set, and FOLL_PIN and FOLL_LONGTERM are set. See get_user_pages_remote() for
• • documentation on the function arguments, because the arguments here are
• • identical.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for further details.
• • FOLL_LONGTERM means that the pages are being pinned for "long term" use,
• • typically by a non-CPU device, and we cannot be sure that waiting for a
• • pinned page to become unpin will be effective.
• • This is intended for Case 2 (RDMA: long-term pins) in
• • Documentation/vm/pin_user_pages.rst.
• */

+long pin_longterm_pages_remote(struct task_struct *tsk, struct mm_struct *mm,

• 	       unsigned long start, unsigned long nr_pages,
  

• 	       unsigned int gup_flags, struct page **pages,
  

• 	       struct vm_area_struct **vmas, int *locked)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= FOLL_LONGTERM | FOLL_REMOTE | FOLL_PIN;
• return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas,

• 		       locked, gup_flags);
  

+} +EXPORT_SYMBOL(pin_longterm_pages_remote);

+/**

• • pin_user_pages() - pin user pages in memory for use by other devices
• • Nearly the same as get_user_pages(), except that FOLL_TOUCH is not set, and
• • FOLL_PIN is set.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for details.
• • This is intended for Case 1 (DIO) in Documentation/vm/pin_user_pages.rst. It
• • is NOT intended for Case 2 (RDMA: long-term pins).
• */

+long pin_user_pages(unsigned long start, unsigned long nr_pages,

•     unsigned int gup_flags, struct page **pages,
  

•     struct vm_area_struct **vmas)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= FOLL_PIN;
• return __gup_longterm_locked(current, current->mm, start, nr_pages,

• 		     pages, vmas, gup_flags);
  

+} +EXPORT_SYMBOL(pin_user_pages);

+/**

• • pin_longterm_pages() - pin user pages in memory for long-term use (RDMA,
• • typically)
• • Nearly the same as get_user_pages(), except that FOLL_PIN and FOLL_LONGTERM
• • are set. See get_user_pages_fast() for documentation on the function
• • arguments, because the arguments here are identical.
• • FOLL_PIN means that the pages must be released via put_user_page(). Please
• • see Documentation/vm/pin_user_pages.rst for further details.
• • FOLL_LONGTERM means that the pages are being pinned for "long term" use,
• • typically by a non-CPU device, and we cannot be sure that waiting for a
• • pinned page to become unpin will be effective.
• • This is intended for Case 2 (RDMA: long-term pins) in
• • Documentation/vm/pin_user_pages.rst.
• */

+long pin_longterm_pages(unsigned long start, unsigned long nr_pages,

• 	unsigned int gup_flags, struct page **pages,
  

• 	struct vm_area_struct **vmas)
  

+{

• /* FOLL_GET and FOLL_PIN are mutually exclusive. */
• if (WARN_ON_ONCE(gup_flags & FOLL_GET))

• return -EINVAL;
  

• gup_flags |= FOLL_PIN | FOLL_LONGTERM;
• return __gup_longterm_locked(current, current->mm, start, nr_pages,

• 		     pages, vmas, gup_flags);
  

+}

+EXPORT_SYMBOL(pin_longterm_pages);

2.24.0