On 2/18/21 9:34 AM, Mike Kravetz wrote:
On 2/18/21 9:25 AM, Jason Gunthorpe wrote:
On Thu, Feb 18, 2021 at 02:45:54PM +0000, Matthew Wilcox wrote:
On Wed, Feb 17, 2021 at 11:02:52AM -0800, Andrew Morton wrote:
On Wed, 17 Feb 2021 10:49:25 -0800 Mike Kravetz mike.kravetz@oracle.com wrote:
page structs are not guaranteed to be contiguous for gigantic pages. The
June 2014. That's a long lurk time for a bug. I wonder if some later commit revealed it.
I would suggest that gigantic pages have not seen much use. Certainly performance with Intel CPUs on benchmarks that I've been involved with showed lower performance with 1GB pages than with 2MB pages until quite recently.
I suggested in another thread that maybe it is time to consider dropping this "feature"
If it has been slightly broken for 7 years it seems a good bet it isn't actually being used.
The cost to fix GUP to be compatible with this will hurt normal GUP performance - and again, that nobody has hit this bug in GUP further suggests the feature isn't used..
I was thinking that we could detect these 'unusual' configurations and only do the slower page struct walking in those cases. However, we would need to do some research to make sure we have taken into account all possible config options which can produce non-contiguous page structs. That should have zero performance impact in the 'normal' cases.
What about something like the following patch, and making all code that wants to scan gigantic page subpages use mem_map_next()?
From 95b0384bd5d7f0435546bdd3c01c478724ae0166 Mon Sep 17 00:00:00 2001
From: Mike Kravetz mike.kravetz@oracle.com Date: Thu, 18 Feb 2021 13:35:02 -0800 Subject: [PATCH] mm: define PFN_PAGE_MAP_LINEAR to optimize gigantic page scans
Signed-off-by: Mike Kravetz mike.kravetz@oracle.com --- arch/ia64/include/asm/page.h | 1 + arch/m68k/include/asm/page_no.h | 1 + include/asm-generic/memory_model.h | 2 ++ mm/internal.h | 2 ++ 4 files changed, 6 insertions(+)
diff --git a/arch/ia64/include/asm/page.h b/arch/ia64/include/asm/page.h index b69a5499d75b..8f4288862ec8 100644 --- a/arch/ia64/include/asm/page.h +++ b/arch/ia64/include/asm/page.h @@ -106,6 +106,7 @@ extern struct page *vmem_map; #ifdef CONFIG_DISCONTIGMEM # define page_to_pfn(page) ((unsigned long) (page - vmem_map)) # define pfn_to_page(pfn) (vmem_map + (pfn)) +# define PFN_PAGE_MAP_LINEAR # define __pfn_to_phys(pfn) PFN_PHYS(pfn) #else # include <asm-generic/memory_model.h> diff --git a/arch/m68k/include/asm/page_no.h b/arch/m68k/include/asm/page_no.h index 6bbe52025de3..cafc0731a42c 100644 --- a/arch/m68k/include/asm/page_no.h +++ b/arch/m68k/include/asm/page_no.h @@ -28,6 +28,7 @@ extern unsigned long memory_end;
#define pfn_to_page(pfn) virt_to_page(pfn_to_virt(pfn)) #define page_to_pfn(page) virt_to_pfn(page_to_virt(page)) +#define PFN_PAGE_MAP_LINEAR #define pfn_valid(pfn) ((pfn) < max_mapnr)
#define virt_addr_valid(kaddr) (((void *)(kaddr) >= (void *)PAGE_OFFSET) && \ diff --git a/include/asm-generic/memory_model.h b/include/asm-generic/memory_model.h index 7637fb46ba4f..8ac4c48dbf22 100644 --- a/include/asm-generic/memory_model.h +++ b/include/asm-generic/memory_model.h @@ -33,6 +33,7 @@ #define __pfn_to_page(pfn) (mem_map + ((pfn) - ARCH_PFN_OFFSET)) #define __page_to_pfn(page) ((unsigned long)((page) - mem_map) + \ ARCH_PFN_OFFSET) +#define PFN_PAGE_MAP_LINEAR #elif defined(CONFIG_DISCONTIGMEM)
#define __pfn_to_page(pfn) \ @@ -53,6 +54,7 @@ /* memmap is virtually contiguous. */ #define __pfn_to_page(pfn) (vmemmap + (pfn)) #define __page_to_pfn(page) (unsigned long)((page) - vmemmap) +#define PFN_PAGE_MAP_LINEAR
#elif defined(CONFIG_SPARSEMEM) /* diff --git a/mm/internal.h b/mm/internal.h index 25d2b2439f19..64cc5069047c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -454,12 +454,14 @@ static inline struct page *mem_map_offset(struct page *base, int offset) static inline struct page *mem_map_next(struct page *iter, struct page *base, int offset) { +#ifndef PFN_PAGE_MAP_LINEAR if (unlikely((offset & (MAX_ORDER_NR_PAGES - 1)) == 0)) { unsigned long pfn = page_to_pfn(base) + offset; if (!pfn_valid(pfn)) return NULL; return pfn_to_page(pfn); } +#endif return iter + 1; }