On 6/3/21 4:59 PM, Mina Almasry wrote:
On Wed, Jun 2, 2021 at 4:50 PM Mike Kravetz mike.kravetz@oracle.com wrote:
The routine restore_reserve_on_error is called to restore reservation information when an error occurs after page allocation. The routine alloc_huge_page modifies the mapping reserve map and potentially the reserve count during allocation. If code calling alloc_huge_page encounters an error after allocation and needs to free the page, the reservation information needs to be adjusted.
Currently, restore_reserve_on_error only takes action on pages for which the reserve count was adjusted(HPageRestoreReserve flag). There is nothing wrong with these adjustments. However, alloc_huge_page ALWAYS modifies the reserve map during allocation even if the reserve count is not adjusted. This can cause issues as observed during development of this patch [1].
One specific series of operations causing an issue is:
- Create a shared hugetlb mapping Reservations for all pages created by default
- Fault in a page in the mapping Reservation exists so reservation count is decremented
- Punch a hole in the file/mapping at index previously faulted Reservation and any associated pages will be removed
- Allocate a page to fill the hole No reservation entry, so reserve count unmodified Reservation entry added to map by alloc_huge_page
- Error after allocation and before instantiating the page Reservation entry remains in map
- Allocate a page to fill the hole Reservation entry exists, so decrement reservation count
This will cause a reservation count underflow as the reservation count was decremented twice for the same index.
A user would observe a very large number for HugePages_Rsvd in /proc/meminfo. This would also likely cause subsequent allocations of hugetlb pages to fail as it would 'appear' that all pages are reserved.
This sequence of operations is unlikely to happen, however they were easily reproduced and observed using hacked up code as described in [1].
Address the issue by having the routine restore_reserve_on_error take action on pages where HPageRestoreReserve is not set. In this case, we need to remove any reserve map entry created by alloc_huge_page. A new helper routine vma_del_reservation assists with this operation.
There are three callers of alloc_huge_page which do not currently call restore_reserve_on error before freeing a page on error paths. Add those missing calls.
[1] https://lore.kernel.org/linux-mm/20210528005029.88088-1-almasrymina@google.c... Fixes: 96b96a96ddee ("mm/hugetlb: fix huge page reservation leak in private mapping error paths" Signed-off-by: Mike Kravetz mike.kravetz@oracle.com Cc: stable@vger.kernel.org
Yep, works perfectly. Thanks!
Thank you for testing!
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h @@ -2360,25 +2376,39 @@ static long vma_add_reservation(struct hstate *h, return __vma_reservation_common(h, vma, addr, VMA_ADD_RESV); }
+static long vma_del_reservation(struct hstate *h,
struct vm_area_struct *vma, unsigned long addr)+{
return __vma_reservation_common(h, vma, addr, VMA_DEL_RESV);+}
/*
- This routine is called to restore a reservation on error paths. In the
- specific error paths, a huge page was allocated (via alloc_huge_page)
- and is about to be freed. If a reservation for the page existed,
- alloc_huge_page would have consumed the reservation and set
- HPageRestoreReserve in the newly allocated page. When the page is freed
- via free_huge_page, the global reservation count will be incremented if
- HPageRestoreReserve is set. However, free_huge_page can not adjust the
- reserve map. Adjust the reserve map here to be consistent with global
- reserve count adjustments to be made by free_huge_page.
- This routine is called to restore reservation information on error paths.
- It should ONLY be called for pages allocated via alloc_huge_page(), and
- the hugetlb mutex should remain held when calling this routine.
- It handles two specific cases:
- A reservation was in place and page consumed the reservation.
- HPageRestoreRsvCnt is set in the page.
HPageRestoreReserve, not HPageRestoreRsvCnt.
Oops, that was from a previous attempt at fixing where I renamed the flag.
I took some time to think about exactly what was needed in error paths after page allocation. The result was this patch. I tried to add lots of comments describing what is being done and why. This code is very complicated with subtle details. The opposite meaning of entries in the reserve map for shared and private mappings being one example.
- No reservation was in place for the page, so HPageRestoreRsvCnt is
Same.
Will be fixed in v2.
Otherwise it looks good to me. Thanks!
Reviewed-by: Mina Almasry almasrymina@google.com
Thank you,