SHMEM_HUGE_DENY is for emergency use by the admin, to disable allocation of shmem huge pages if, for example, a dangerous bug is found in their usage: see "deny" in Documentation/mm/transhuge.rst. An app using madvise(,,MADV_COLLAPSE) should not be allowed to override it: restore its precedence over shmem_huge_force.
Restore SHMEM_HUGE_DENY precedence over MADV_COLLAPSE.
Fixes: 7c6c6cc4d3a2 ("mm/shmem: add flag to enforce shmem THP in hugepage_vma_check()") Suggested-by: Hugh Dickins hughd@google.com Signed-off-by: Zach O'Keefe zokeefe@google.com Cc: Yang Shi shy828301@gmail.com Cc: stable@vger.kernel.org --- v2->v3: Add 'Cc: stable@vger.kernel.org' as per stable-kernel-rules. v1->v2: Update changelog, and add note explaining rationale for backporting (Andrew Morton).
Request to backport this to 6.1.X stable. We'd like SHMEM_HUGE_DENY to take precedence over MADV_COLLAPSE. If we make this change later, it will be a userspace API change. As such, 6.1 cannot be allowed to continue as-is, and we should fix up the code there. --- mm/shmem.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c index c301487be5fb..0005ab2c29af 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -478,12 +478,10 @@ bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode, if (vma && ((vma->vm_flags & VM_NOHUGEPAGE) || test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags))) return false; - if (shmem_huge_force) - return true; - if (shmem_huge == SHMEM_HUGE_FORCE) - return true; if (shmem_huge == SHMEM_HUGE_DENY) return false; + if (shmem_huge_force || shmem_huge == SHMEM_HUGE_FORCE) + return true;
switch (SHMEM_SB(inode->i_sb)->huge) { case SHMEM_HUGE_ALWAYS: