September 2025 - Linux-stable-mirror

FAILED: patch "[PATCH] mm: introduce and use {pgd,p4d}_populate_kernel()" failed to apply to 6.6-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 6.6-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. To reproduce the conflict and resubmit, you may use the following commands: git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.6.y git checkout FETCH_HEAD git cherry-pick -x f2d2f9598ebb0158a3fe17cda0106d7752e654a2 # <resolve conflicts, build, test, etc.> git commit -s git send-email --to '<stable(a)vger.kernel.org>' --in-reply-to '2025090604-obnoxious-bronco-1690@gregkh' --subject-prefix 'PATCH 6.6.y' HEAD^.. Possible dependencies: thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From f2d2f9598ebb0158a3fe17cda0106d7752e654a2 Mon Sep 17 00:00:00 2001 From: Harry Yoo <harry.yoo(a)oracle.com> Date: Mon, 18 Aug 2025 11:02:05 +0900 Subject: [PATCH] mm: introduce and use {pgd,p4d}_populate_kernel() Introduce and use {pgd,p4d}_populate_kernel() in core MM code when populating PGD and P4D entries for the kernel address space. These helpers ensure proper synchronization of page tables when updating the kernel portion of top-level page tables. Until now, the kernel has relied on each architecture to handle synchronization of top-level page tables in an ad-hoc manner. For example, see commit 9b861528a801 ("x86-64, mem: Update all PGDs for direct mapping and vmemmap mapping changes"). However, this approach has proven fragile for following reasons: 1) It is easy to forget to perform the necessary page table synchronization when introducing new changes. For instance, commit 4917f55b4ef9 ("mm/sparse-vmemmap: improve memory savings for compound devmaps") overlooked the need to synchronize page tables for the vmemmap area. 2) It is also easy to overlook that the vmemmap and direct mapping areas must not be accessed before explicit page table synchronization. For example, commit 8d400913c231 ("x86/vmemmap: handle unpopulated sub-pmd ranges")) caused crashes by accessing the vmemmap area before calling sync_global_pgds(). To address this, as suggested by Dave Hansen, introduce _kernel() variants of the page table population helpers, which invoke architecture-specific hooks to properly synchronize page tables. These are introduced in a new header file, include/linux/pgalloc.h, so they can be called from common code. They reuse existing infrastructure for vmalloc and ioremap. Synchronization requirements are determined by ARCH_PAGE_TABLE_SYNC_MASK, and the actual synchronization is performed by arch_sync_kernel_mappings(). This change currently targets only x86_64, so only PGD and P4D level helpers are introduced. Currently, these helpers are no-ops since no architecture sets PGTBL_{PGD,P4D}_MODIFIED in ARCH_PAGE_TABLE_SYNC_MASK. In theory, PUD and PMD level helpers can be added later if needed by other architectures. For now, 32-bit architectures (x86-32 and arm) only handle PGTBL_PMD_MODIFIED, so p*d_populate_kernel() will never affect them unless we introduce a PMD level helper. [harry.yoo(a)oracle.com: fix KASAN build error due to p*d_populate_kernel()] Link: https://lkml.kernel.org/r/20250822020727.202749-1-harry.yoo@oracle.com Link: https://lkml.kernel.org/r/20250818020206.4517-3-harry.yoo@oracle.com Fixes: 8d400913c231 ("x86/vmemmap: handle unpopulated sub-pmd ranges") Signed-off-by: Harry Yoo <harry.yoo(a)oracle.com> Suggested-by: Dave Hansen <dave.hansen(a)linux.intel.com> Acked-by: Kiryl Shutsemau <kas(a)kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt(a)kernel.org> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes(a)oracle.com> Acked-by: David Hildenbrand <david(a)redhat.com> Cc: Alexander Potapenko <glider(a)google.com> Cc: Alistair Popple <apopple(a)nvidia.com> Cc: Andrey Konovalov <andreyknvl(a)gmail.com> Cc: Andrey Ryabinin <ryabinin.a.a(a)gmail.com> Cc: Andy Lutomirski <luto(a)kernel.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Anshuman Khandual <anshuman.khandual(a)arm.com> Cc: Ard Biesheuvel <ardb(a)kernel.org> Cc: Arnd Bergmann <arnd(a)arndb.de> Cc: bibo mao <maobibo(a)loongson.cn> Cc: Borislav Betkov <bp(a)alien8.de> Cc: Christoph Lameter (Ampere) <cl(a)gentwo.org> Cc: Dennis Zhou <dennis(a)kernel.org> Cc: Dev Jain <dev.jain(a)arm.com> Cc: Dmitriy Vyukov <dvyukov(a)google.com> Cc: Gwan-gyeong Mun <gwan-gyeong.mun(a)intel.com> Cc: Ingo Molnar <mingo(a)redhat.com> Cc: Jane Chu <jane.chu(a)oracle.com> Cc: Joao Martins <joao.m.martins(a)oracle.com> Cc: Joerg Roedel <joro(a)8bytes.org> Cc: John Hubbard <jhubbard(a)nvidia.com> Cc: Kevin Brodsky <kevin.brodsky(a)arm.com> Cc: Liam Howlett <liam.howlett(a)oracle.com> Cc: Michal Hocko <mhocko(a)suse.com> Cc: Oscar Salvador <osalvador(a)suse.de> Cc: Peter Xu <peterx(a)redhat.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Qi Zheng <zhengqi.arch(a)bytedance.com> Cc: Ryan Roberts <ryan.roberts(a)arm.com> Cc: Suren Baghdasaryan <surenb(a)google.com> Cc: Tejun Heo <tj(a)kernel.org> Cc: Thomas Gleinxer <tglx(a)linutronix.de> Cc: Thomas Huth <thuth(a)redhat.com> Cc: "Uladzislau Rezki (Sony)" <urezki(a)gmail.com> Cc: Vincenzo Frascino <vincenzo.frascino(a)arm.com> Cc: Vlastimil Babka <vbabka(a)suse.cz> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h new file mode 100644 index 000000000000..9174fa59bbc5 --- /dev/null +++ b/include/linux/pgalloc.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PGALLOC_H +#define _LINUX_PGALLOC_H + +#include <linux/pgtable.h> +#include <asm/pgalloc.h> + +/* + * {pgd,p4d}_populate_kernel() are defined as macros to allow + * compile-time optimization based on the configured page table levels. + * Without this, linking may fail because callers (e.g., KASAN) may rely + * on calls to these functions being optimized away when passing symbols + * that exist only for certain page table levels. + */ +#define pgd_populate_kernel(addr, pgd, p4d) \ + do { \ + pgd_populate(&init_mm, pgd, p4d); \ + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_PGD_MODIFIED) \ + arch_sync_kernel_mappings(addr, addr); \ + } while (0) + +#define p4d_populate_kernel(addr, p4d, pud) \ + do { \ + p4d_populate(&init_mm, p4d, pud); \ + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_P4D_MODIFIED) \ + arch_sync_kernel_mappings(addr, addr); \ + } while (0) + +#endif /* _LINUX_PGALLOC_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index ba699df6ef69..2b80fd456c8b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1469,8 +1469,8 @@ static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values - * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() - * needs to be called. + * and let generic vmalloc, ioremap and page table update code know when + * arch_sync_kernel_mappings() needs to be called. */ #ifndef ARCH_PAGE_TABLE_SYNC_MASK #define ARCH_PAGE_TABLE_SYNC_MASK 0 @@ -1954,10 +1954,11 @@ static inline bool arch_has_pfn_modify_check(void) /* * Page Table Modification bits for pgtbl_mod_mask. * - * These are used by the p?d_alloc_track*() set of functions an in the generic - * vmalloc/ioremap code to track at which page-table levels entries have been - * modified. Based on that the code can better decide when vmalloc and ioremap - * mapping changes need to be synchronized to other page-tables in the system. + * These are used by the p?d_alloc_track*() and p*d_populate_kernel() + * functions in the generic vmalloc, ioremap and page table update code + * to track at which page-table levels entries have been modified. + * Based on that the code can better decide when page table changes need + * to be synchronized to other page-tables in the system. */ #define __PGTBL_PGD_MODIFIED 0 #define __PGTBL_P4D_MODIFIED 1 diff --git a/mm/kasan/init.c b/mm/kasan/init.c index ced6b29fcf76..8fce3370c84e 100644 --- a/mm/kasan/init.c +++ b/mm/kasan/init.c @@ -13,9 +13,9 @@ #include <linux/mm.h> #include <linux/pfn.h> #include <linux/slab.h> +#include <linux/pgalloc.h> #include <asm/page.h> -#include <asm/pgalloc.h> #include "kasan.h" @@ -191,7 +191,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, pud_t *pud; pmd_t *pmd; - p4d_populate(&init_mm, p4d, + p4d_populate_kernel(addr, p4d, lm_alias(kasan_early_shadow_pud)); pud = pud_offset(p4d, addr); pud_populate(&init_mm, pud, @@ -212,7 +212,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, } else { p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); pud_init(p); - p4d_populate(&init_mm, p4d, p); + p4d_populate_kernel(addr, p4d, p); } } zero_pud_populate(p4d, addr, next); @@ -251,10 +251,10 @@ int __ref kasan_populate_early_shadow(const void *shadow_start, * puds,pmds, so pgd_populate(), pud_populate() * is noops. */ - pgd_populate(&init_mm, pgd, + pgd_populate_kernel(addr, pgd, lm_alias(kasan_early_shadow_p4d)); p4d = p4d_offset(pgd, addr); - p4d_populate(&init_mm, p4d, + p4d_populate_kernel(addr, p4d, lm_alias(kasan_early_shadow_pud)); pud = pud_offset(p4d, addr); pud_populate(&init_mm, pud, @@ -273,7 +273,7 @@ int __ref kasan_populate_early_shadow(const void *shadow_start, if (!p) return -ENOMEM; } else { - pgd_populate(&init_mm, pgd, + pgd_populate_kernel(addr, pgd, early_alloc(PAGE_SIZE, NUMA_NO_NODE)); } } diff --git a/mm/percpu.c b/mm/percpu.c index d9cbaee92b60..a56f35dcc417 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3108,7 +3108,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, #endif /* BUILD_EMBED_FIRST_CHUNK */ #ifdef BUILD_PAGE_FIRST_CHUNK -#include <asm/pgalloc.h> +#include <linux/pgalloc.h> #ifndef P4D_TABLE_SIZE #define P4D_TABLE_SIZE PAGE_SIZE @@ -3134,13 +3134,13 @@ void __init __weak pcpu_populate_pte(unsigned long addr) if (pgd_none(*pgd)) { p4d = memblock_alloc_or_panic(P4D_TABLE_SIZE, P4D_TABLE_SIZE); - pgd_populate(&init_mm, pgd, p4d); + pgd_populate_kernel(addr, pgd, p4d); } p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) { pud = memblock_alloc_or_panic(PUD_TABLE_SIZE, PUD_TABLE_SIZE); - p4d_populate(&init_mm, p4d, pud); + p4d_populate_kernel(addr, p4d, pud); } pud = pud_offset(p4d, addr); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 41aa0493eb03..dbd8daccade2 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -27,9 +27,9 @@ #include <linux/spinlock.h> #include <linux/vmalloc.h> #include <linux/sched.h> +#include <linux/pgalloc.h> #include <asm/dma.h> -#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include "hugetlb_vmemmap.h" @@ -229,7 +229,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node) if (!p) return NULL; pud_init(p); - p4d_populate(&init_mm, p4d, p); + p4d_populate_kernel(addr, p4d, p); } return p4d; } @@ -241,7 +241,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; - pgd_populate(&init_mm, pgd, p); + pgd_populate_kernel(addr, pgd, p); } return pgd; }

5 days, 21 hours

2
3
0 0

FAILED: patch "[PATCH] mm: introduce and use {pgd,p4d}_populate_kernel()" failed to apply to 6.12-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 6.12-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. To reproduce the conflict and resubmit, you may use the following commands: git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.12.y git checkout FETCH_HEAD git cherry-pick -x f2d2f9598ebb0158a3fe17cda0106d7752e654a2 # <resolve conflicts, build, test, etc.> git commit -s git send-email --to '<stable(a)vger.kernel.org>' --in-reply-to '2025090602-bullwhip-runner-63fe@gregkh' --subject-prefix 'PATCH 6.12.y' HEAD^.. Possible dependencies: thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From f2d2f9598ebb0158a3fe17cda0106d7752e654a2 Mon Sep 17 00:00:00 2001 From: Harry Yoo <harry.yoo(a)oracle.com> Date: Mon, 18 Aug 2025 11:02:05 +0900 Subject: [PATCH] mm: introduce and use {pgd,p4d}_populate_kernel() Introduce and use {pgd,p4d}_populate_kernel() in core MM code when populating PGD and P4D entries for the kernel address space. These helpers ensure proper synchronization of page tables when updating the kernel portion of top-level page tables. Until now, the kernel has relied on each architecture to handle synchronization of top-level page tables in an ad-hoc manner. For example, see commit 9b861528a801 ("x86-64, mem: Update all PGDs for direct mapping and vmemmap mapping changes"). However, this approach has proven fragile for following reasons: 1) It is easy to forget to perform the necessary page table synchronization when introducing new changes. For instance, commit 4917f55b4ef9 ("mm/sparse-vmemmap: improve memory savings for compound devmaps") overlooked the need to synchronize page tables for the vmemmap area. 2) It is also easy to overlook that the vmemmap and direct mapping areas must not be accessed before explicit page table synchronization. For example, commit 8d400913c231 ("x86/vmemmap: handle unpopulated sub-pmd ranges")) caused crashes by accessing the vmemmap area before calling sync_global_pgds(). To address this, as suggested by Dave Hansen, introduce _kernel() variants of the page table population helpers, which invoke architecture-specific hooks to properly synchronize page tables. These are introduced in a new header file, include/linux/pgalloc.h, so they can be called from common code. They reuse existing infrastructure for vmalloc and ioremap. Synchronization requirements are determined by ARCH_PAGE_TABLE_SYNC_MASK, and the actual synchronization is performed by arch_sync_kernel_mappings(). This change currently targets only x86_64, so only PGD and P4D level helpers are introduced. Currently, these helpers are no-ops since no architecture sets PGTBL_{PGD,P4D}_MODIFIED in ARCH_PAGE_TABLE_SYNC_MASK. In theory, PUD and PMD level helpers can be added later if needed by other architectures. For now, 32-bit architectures (x86-32 and arm) only handle PGTBL_PMD_MODIFIED, so p*d_populate_kernel() will never affect them unless we introduce a PMD level helper. [harry.yoo(a)oracle.com: fix KASAN build error due to p*d_populate_kernel()] Link: https://lkml.kernel.org/r/20250822020727.202749-1-harry.yoo@oracle.com Link: https://lkml.kernel.org/r/20250818020206.4517-3-harry.yoo@oracle.com Fixes: 8d400913c231 ("x86/vmemmap: handle unpopulated sub-pmd ranges") Signed-off-by: Harry Yoo <harry.yoo(a)oracle.com> Suggested-by: Dave Hansen <dave.hansen(a)linux.intel.com> Acked-by: Kiryl Shutsemau <kas(a)kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt(a)kernel.org> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes(a)oracle.com> Acked-by: David Hildenbrand <david(a)redhat.com> Cc: Alexander Potapenko <glider(a)google.com> Cc: Alistair Popple <apopple(a)nvidia.com> Cc: Andrey Konovalov <andreyknvl(a)gmail.com> Cc: Andrey Ryabinin <ryabinin.a.a(a)gmail.com> Cc: Andy Lutomirski <luto(a)kernel.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Anshuman Khandual <anshuman.khandual(a)arm.com> Cc: Ard Biesheuvel <ardb(a)kernel.org> Cc: Arnd Bergmann <arnd(a)arndb.de> Cc: bibo mao <maobibo(a)loongson.cn> Cc: Borislav Betkov <bp(a)alien8.de> Cc: Christoph Lameter (Ampere) <cl(a)gentwo.org> Cc: Dennis Zhou <dennis(a)kernel.org> Cc: Dev Jain <dev.jain(a)arm.com> Cc: Dmitriy Vyukov <dvyukov(a)google.com> Cc: Gwan-gyeong Mun <gwan-gyeong.mun(a)intel.com> Cc: Ingo Molnar <mingo(a)redhat.com> Cc: Jane Chu <jane.chu(a)oracle.com> Cc: Joao Martins <joao.m.martins(a)oracle.com> Cc: Joerg Roedel <joro(a)8bytes.org> Cc: John Hubbard <jhubbard(a)nvidia.com> Cc: Kevin Brodsky <kevin.brodsky(a)arm.com> Cc: Liam Howlett <liam.howlett(a)oracle.com> Cc: Michal Hocko <mhocko(a)suse.com> Cc: Oscar Salvador <osalvador(a)suse.de> Cc: Peter Xu <peterx(a)redhat.com> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Qi Zheng <zhengqi.arch(a)bytedance.com> Cc: Ryan Roberts <ryan.roberts(a)arm.com> Cc: Suren Baghdasaryan <surenb(a)google.com> Cc: Tejun Heo <tj(a)kernel.org> Cc: Thomas Gleinxer <tglx(a)linutronix.de> Cc: Thomas Huth <thuth(a)redhat.com> Cc: "Uladzislau Rezki (Sony)" <urezki(a)gmail.com> Cc: Vincenzo Frascino <vincenzo.frascino(a)arm.com> Cc: Vlastimil Babka <vbabka(a)suse.cz> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h new file mode 100644 index 000000000000..9174fa59bbc5 --- /dev/null +++ b/include/linux/pgalloc.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PGALLOC_H +#define _LINUX_PGALLOC_H + +#include <linux/pgtable.h> +#include <asm/pgalloc.h> + +/* + * {pgd,p4d}_populate_kernel() are defined as macros to allow + * compile-time optimization based on the configured page table levels. + * Without this, linking may fail because callers (e.g., KASAN) may rely + * on calls to these functions being optimized away when passing symbols + * that exist only for certain page table levels. + */ +#define pgd_populate_kernel(addr, pgd, p4d) \ + do { \ + pgd_populate(&init_mm, pgd, p4d); \ + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_PGD_MODIFIED) \ + arch_sync_kernel_mappings(addr, addr); \ + } while (0) + +#define p4d_populate_kernel(addr, p4d, pud) \ + do { \ + p4d_populate(&init_mm, p4d, pud); \ + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_P4D_MODIFIED) \ + arch_sync_kernel_mappings(addr, addr); \ + } while (0) + +#endif /* _LINUX_PGALLOC_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index ba699df6ef69..2b80fd456c8b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1469,8 +1469,8 @@ static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values - * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() - * needs to be called. + * and let generic vmalloc, ioremap and page table update code know when + * arch_sync_kernel_mappings() needs to be called. */ #ifndef ARCH_PAGE_TABLE_SYNC_MASK #define ARCH_PAGE_TABLE_SYNC_MASK 0 @@ -1954,10 +1954,11 @@ static inline bool arch_has_pfn_modify_check(void) /* * Page Table Modification bits for pgtbl_mod_mask. * - * These are used by the p?d_alloc_track*() set of functions an in the generic - * vmalloc/ioremap code to track at which page-table levels entries have been - * modified. Based on that the code can better decide when vmalloc and ioremap - * mapping changes need to be synchronized to other page-tables in the system. + * These are used by the p?d_alloc_track*() and p*d_populate_kernel() + * functions in the generic vmalloc, ioremap and page table update code + * to track at which page-table levels entries have been modified. + * Based on that the code can better decide when page table changes need + * to be synchronized to other page-tables in the system. */ #define __PGTBL_PGD_MODIFIED 0 #define __PGTBL_P4D_MODIFIED 1 diff --git a/mm/kasan/init.c b/mm/kasan/init.c index ced6b29fcf76..8fce3370c84e 100644 --- a/mm/kasan/init.c +++ b/mm/kasan/init.c @@ -13,9 +13,9 @@ #include <linux/mm.h> #include <linux/pfn.h> #include <linux/slab.h> +#include <linux/pgalloc.h> #include <asm/page.h> -#include <asm/pgalloc.h> #include "kasan.h" @@ -191,7 +191,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, pud_t *pud; pmd_t *pmd; - p4d_populate(&init_mm, p4d, + p4d_populate_kernel(addr, p4d, lm_alias(kasan_early_shadow_pud)); pud = pud_offset(p4d, addr); pud_populate(&init_mm, pud, @@ -212,7 +212,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, } else { p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); pud_init(p); - p4d_populate(&init_mm, p4d, p); + p4d_populate_kernel(addr, p4d, p); } } zero_pud_populate(p4d, addr, next); @@ -251,10 +251,10 @@ int __ref kasan_populate_early_shadow(const void *shadow_start, * puds,pmds, so pgd_populate(), pud_populate() * is noops. */ - pgd_populate(&init_mm, pgd, + pgd_populate_kernel(addr, pgd, lm_alias(kasan_early_shadow_p4d)); p4d = p4d_offset(pgd, addr); - p4d_populate(&init_mm, p4d, + p4d_populate_kernel(addr, p4d, lm_alias(kasan_early_shadow_pud)); pud = pud_offset(p4d, addr); pud_populate(&init_mm, pud, @@ -273,7 +273,7 @@ int __ref kasan_populate_early_shadow(const void *shadow_start, if (!p) return -ENOMEM; } else { - pgd_populate(&init_mm, pgd, + pgd_populate_kernel(addr, pgd, early_alloc(PAGE_SIZE, NUMA_NO_NODE)); } } diff --git a/mm/percpu.c b/mm/percpu.c index d9cbaee92b60..a56f35dcc417 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3108,7 +3108,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, #endif /* BUILD_EMBED_FIRST_CHUNK */ #ifdef BUILD_PAGE_FIRST_CHUNK -#include <asm/pgalloc.h> +#include <linux/pgalloc.h> #ifndef P4D_TABLE_SIZE #define P4D_TABLE_SIZE PAGE_SIZE @@ -3134,13 +3134,13 @@ void __init __weak pcpu_populate_pte(unsigned long addr) if (pgd_none(*pgd)) { p4d = memblock_alloc_or_panic(P4D_TABLE_SIZE, P4D_TABLE_SIZE); - pgd_populate(&init_mm, pgd, p4d); + pgd_populate_kernel(addr, pgd, p4d); } p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) { pud = memblock_alloc_or_panic(PUD_TABLE_SIZE, PUD_TABLE_SIZE); - p4d_populate(&init_mm, p4d, pud); + p4d_populate_kernel(addr, p4d, pud); } pud = pud_offset(p4d, addr); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 41aa0493eb03..dbd8daccade2 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -27,9 +27,9 @@ #include <linux/spinlock.h> #include <linux/vmalloc.h> #include <linux/sched.h> +#include <linux/pgalloc.h> #include <asm/dma.h> -#include <asm/pgalloc.h> #include <asm/tlbflush.h> #include "hugetlb_vmemmap.h" @@ -229,7 +229,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node) if (!p) return NULL; pud_init(p); - p4d_populate(&init_mm, p4d, p); + p4d_populate_kernel(addr, p4d, p); } return p4d; } @@ -241,7 +241,7 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; - pgd_populate(&init_mm, pgd, p); + pgd_populate_kernel(addr, pgd, p); } return pgd; }

5 days, 21 hours

3
9
0 0

[PATCH v1] rust: cfi: only 64-bit arm and x86 support CFI_CLANG

by Conor Dooley

From: Conor Dooley <conor.dooley(a)microchip.com> The kernel uses the standard rustc targets for non-x86 targets, and out of those only 64-bit arm's target has kcfi support enabled. For x86, the custom 64-bit target enables kcfi. The HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC config option that allows CFI_CLANG to be used in combination with RUST does not check whether the rustc target supports kcfi. This breaks the build on riscv (and presumably 32-bit arm) when CFI_CLANG and RUST are enabled at the same time. Ordinarily, a rustc-option check would be used to detect target support but unfortunately rustc-option filters out the target for reasons given in commit 46e24a545cdb4 ("rust: kasan/kbuild: fix missing flags on first build"). As a result, if the host supports kcfi but the target does not, e.g. when building for riscv on x86_64, the build would remain broken. Instead, make HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC depend on the only two architectures where the target used supports it to fix the build. CC: stable(a)vger.kernel.org Fixes: ca627e636551e ("rust: cfi: add support for CFI_CLANG with Rust") Signed-off-by: Conor Dooley <conor.dooley(a)microchip.com> --- CC: Paul Walmsley <paul.walmsley(a)sifive.com> CC: Palmer Dabbelt <palmer(a)dabbelt.com> CC: Alexandre Ghiti <alex(a)ghiti.fr> CC: Miguel Ojeda <ojeda(a)kernel.org> CC: Alex Gaynor <alex.gaynor(a)gmail.com> CC: Boqun Feng <boqun.feng(a)gmail.com> CC: Gary Guo <gary(a)garyguo.net> CC: "Björn Roy Baron" <bjorn3_gh(a)protonmail.com> CC: Benno Lossin <lossin(a)kernel.org> CC: Andreas Hindborg <a.hindborg(a)kernel.org> CC: Alice Ryhl <aliceryhl(a)google.com> CC: Trevor Gross <tmgross(a)umich.edu> CC: Danilo Krummrich <dakr(a)kernel.org> CC: Kees Cook <kees(a)kernel.org> CC: Sami Tolvanen <samitolvanen(a)google.com> CC: Matthew Maurer <mmaurer(a)google.com> CC: "Peter Zijlstra (Intel)" <peterz(a)infradead.org> CC: linux-kernel(a)vger.kernel.org CC: linux-riscv(a)lists.infradead.org CC: rust-for-linux(a)vger.kernel.org --- arch/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/Kconfig b/arch/Kconfig index d1b4ffd6e0856..880cddff5eda7 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -917,6 +917,7 @@ config HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC def_bool y depends on HAVE_CFI_ICALL_NORMALIZE_INTEGERS_CLANG depends on RUSTC_VERSION >= 107900 + depends on ARM64 || X86_64 # With GCOV/KASAN we need this fix: https://github.com/rust-lang/rust/pull/129373 depends on (RUSTC_LLVM_VERSION >= 190103 && RUSTC_VERSION >= 108200) || \ (!GCOV_KERNEL && !KASAN_GENERIC && !KASAN_SW_TAGS) -- 2.47.2

5 days, 21 hours

4
4
0 0

[PATCH rdma-next 1/1] RDMA/core: Fix socket leak in rdma_dev_init_net

by Håkon Bugge

If rdma_dev_init_net() has an early return because the supplied net is the default init_net, we need to call rdma_nl_net_exit() before returning. Fixes: 4e0f7b907072 ("RDMA/core: Implement compat device/sysfs tree in net namespace") Cc: stable(a)vger.kernel.org Signed-off-by: Håkon Bugge <haakon.bugge(a)oracle.com> --- drivers/infiniband/core/device.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 3145cb34a1d20..ec5642e70c5db 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -1203,8 +1203,10 @@ static __net_init int rdma_dev_init_net(struct net *net) return ret; /* No need to create any compat devices in default init_net. */ - if (net_eq(net, &init_net)) + if (net_eq(net, &init_net)) { + rdma_nl_net_exit(rnet); return 0; + } ret = xa_alloc(&rdma_nets, &rnet->id, rnet, xa_limit_32b, GFP_KERNEL); if (ret) { -- 2.43.5

5 days, 22 hours

4
3
0 0

Re: [PATCH net v4] netrom: linearize and validate lengths in nr_rx_frame()

by Jakub Kicinski

On Sun, 7 Sep 2025 16:32:03 +0200 Bernard Pidoux wrote: > While applying netrom PATCH net v4 > patch says that > it is malformed on line 12. FWIW the version I received is completely mangled. There's a leading space before each +. You can try B4 relay if your mail server is giving you grief.

5 days, 23 hours

2
1
0 0

[PATCH v5 02/13] ASoC: codecs: wcd937x: make stub functions inline

by Srinivas Kandagatla

For some reason we ended up with stub functions that are not inline, this can result in build error if its included multiple places, as we will be redefining the same function Fixes: c99a515ff153 ("ASoC: codecs: wcd937x-sdw: add SoundWire driver") Cc: <Stable(a)vger.kernel.org> Signed-off-by: Srinivas Kandagatla <srinivas.kandagatla(a)oss.qualcomm.com> --- sound/soc/codecs/wcd937x.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sound/soc/codecs/wcd937x.h b/sound/soc/codecs/wcd937x.h index 3ab21bb5846e..d20886a2803a 100644 --- a/sound/soc/codecs/wcd937x.h +++ b/sound/soc/codecs/wcd937x.h @@ -552,21 +552,21 @@ int wcd937x_sdw_hw_params(struct wcd937x_sdw_priv *wcd, struct device *wcd937x_sdw_device_get(struct device_node *np); #else -int wcd937x_sdw_free(struct wcd937x_sdw_priv *wcd, +static inline int wcd937x_sdw_free(struct wcd937x_sdw_priv *wcd, struct snd_pcm_substream *substream, struct snd_soc_dai *dai) { return -EOPNOTSUPP; } -int wcd937x_sdw_set_sdw_stream(struct wcd937x_sdw_priv *wcd, +static inline int wcd937x_sdw_set_sdw_stream(struct wcd937x_sdw_priv *wcd, struct snd_soc_dai *dai, void *stream, int direction) { return -EOPNOTSUPP; } -int wcd937x_sdw_hw_params(struct wcd937x_sdw_priv *wcd, +static inline int wcd937x_sdw_hw_params(struct wcd937x_sdw_priv *wcd, struct snd_pcm_substream *substream, struct snd_pcm_hw_params *params, struct snd_soc_dai *dai) -- 2.50.0

5 days, 23 hours

1
0
0 0

FAILED: patch "[PATCH] tracing: Do not add length to print format in synthetic" failed to apply to 6.1-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 6.1-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. To reproduce the conflict and resubmit, you may use the following commands: git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.1.y git checkout FETCH_HEAD git cherry-pick -x e1a453a57bc76be678bd746f84e3d73f378a9511 # <resolve conflicts, build, test, etc.> git commit -s git send-email --to '<stable(a)vger.kernel.org>' --in-reply-to '2025041706-ambiance-zen-5f4e@gregkh' --subject-prefix 'PATCH 6.1.y' HEAD^.. Possible dependencies: thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From e1a453a57bc76be678bd746f84e3d73f378a9511 Mon Sep 17 00:00:00 2001 From: Steven Rostedt <rostedt(a)goodmis.org> Date: Mon, 7 Apr 2025 15:41:39 -0400 Subject: [PATCH] tracing: Do not add length to print format in synthetic events The following causes a vsnprintf fault: # echo 's:wake_lat char[] wakee; u64 delta;' >> /sys/kernel/tracing/dynamic_events # echo 'hist:keys=pid:ts=common_timestamp.usecs if !(common_flags & 0x18)' > /sys/kernel/tracing/events/sched/sched_waking/trigger # echo 'hist:keys=next_pid:delta=common_timestamp.usecs-$ts:onmatch(sched.sched_waking).trace(wake_lat,next_comm,$delta)' > /sys/kernel/tracing/events/sched/sched_switch/trigger Because the synthetic event's "wakee" field is created as a dynamic string (even though the string copied is not). The print format to print the dynamic string changed from "%*s" to "%s" because another location (__set_synth_event_print_fmt()) exported this to user space, and user space did not need that. But it is still used in print_synth_event(), and the output looks like: <idle>-0 [001] d..5. 193.428167: wake_lat: wakee=(efault)sshd-sessiondelta=155 sshd-session-879 [001] d..5. 193.811080: wake_lat: wakee=(efault)kworker/u34:5delta=58 <idle>-0 [002] d..5. 193.811198: wake_lat: wakee=(efault)bashdelta=91 bash-880 [002] d..5. 193.811371: wake_lat: wakee=(efault)kworker/u35:2delta=21 <idle>-0 [001] d..5. 193.811516: wake_lat: wakee=(efault)sshd-sessiondelta=129 sshd-session-879 [001] d..5. 193.967576: wake_lat: wakee=(efault)kworker/u34:5delta=50 The length isn't needed as the string is always nul terminated. Just print the string and not add the length (which was hard coded to the max string length anyway). Cc: stable(a)vger.kernel.org Cc: Mathieu Desnoyers <mathieu.desnoyers(a)efficios.com> Cc: Tom Zanussi <zanussi(a)kernel.org> Cc: Douglas Raillard <douglas.raillard(a)arm.com> Acked-by: Masami Hiramatsu (Google) <mhiramat(a)kernel.org> Link: https://lore.kernel.org/20250407154139.69955768@gandalf.local.home Fixes: 4d38328eb442d ("tracing: Fix synth event printk format for str fields"); Signed-off-by: Steven Rostedt (Google) <rostedt(a)goodmis.org> diff --git a/kernel/trace/trace_events_synth.c b/kernel/trace/trace_events_synth.c index 969f48742d72..33cfbd4ed76d 100644 --- a/kernel/trace/trace_events_synth.c +++ b/kernel/trace/trace_events_synth.c @@ -370,7 +370,6 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter, union trace_synth_field *data = &entry->fields[n_u64]; trace_seq_printf(s, print_fmt, se->fields[i]->name, - STR_VAR_LEN_MAX, (char *)entry + data->as_dynamic.offset, i == se->n_fields - 1 ? "" : " "); n_u64++;

5 days, 23 hours

3
2
0 0

[PATCH v6 1/3] drm/buddy: Optimize free block management with RB tree

by Arunpravin Paneer Selvam

Replace the freelist (O(n)) used for free block management with a red-black tree, providing more efficient O(log n) search, insert, and delete operations. This improves scalability and performance when managing large numbers of free blocks per order (e.g., hundreds or thousands). In the VK-CTS memory stress subtest, the buddy manager merges fragmented memory and inserts freed blocks into the freelist. Since freelist insertion is O(n), this becomes a bottleneck as fragmentation increases. Benchmarking shows list_insert_sorted() consumes ~52.69% CPU with the freelist, compared to just 0.03% with the RB tree (rbtree_insert.isra.0), despite performing the same sorted insert. This also improves performance in heavily fragmented workloads, such as games or graphics tests that stress memory. As the buddy allocator evolves with new features such as clear-page tracking, the resulting fragmentation and complexity have grown. These RB-tree based design changes are introduced to address that growth and ensure the allocator continues to perform efficiently under fragmented conditions. The RB tree implementation with separate clear/dirty trees provides: - O(n log n) aggregate complexity for all operations instead of O(n^2) - Elimination of soft lockups and system instability - Improved code maintainability and clarity - Better scalability for large memory systems - Predictable performance under fragmentation v3(Matthew): - Remove RB_EMPTY_NODE check in force_merge function. - Rename rb for loop macros to have less generic names and move to .c file. - Make the rb node rb and link field as union. v4(Jani Nikula): - The kernel-doc comment should be "/**" - Move all the rbtree macros to rbtree.h and add parens to ensure correct precedence. v5: - Remove the inline in a .c file (Jani Nikula). v6(Peter Zijlstra): - Add rb_add() function replacing the existing rbtree_insert() code. Cc: stable(a)vger.kernel.org Fixes: a68c7eaa7a8f ("drm/amdgpu: Enable clear page functionality") Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam(a)amd.com> --- drivers/gpu/drm/drm_buddy.c | 144 ++++++++++++++++++++++-------------- include/drm/drm_buddy.h | 11 ++- include/linux/rbtree.h | 56 ++++++++++++++ 3 files changed, 153 insertions(+), 58 deletions(-) diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c index a94061f373de..89f4b49ae3fb 100644 --- a/drivers/gpu/drm/drm_buddy.c +++ b/drivers/gpu/drm/drm_buddy.c @@ -14,6 +14,8 @@ static struct kmem_cache *slab_blocks; +#define rbtree_get_free_block(node) rb_entry((node), struct drm_buddy_block, rb) + static struct drm_buddy_block *drm_block_alloc(struct drm_buddy *mm, struct drm_buddy_block *parent, unsigned int order, @@ -31,6 +33,8 @@ static struct drm_buddy_block *drm_block_alloc(struct drm_buddy *mm, block->header |= order; block->parent = parent; + RB_CLEAR_NODE(&block->rb); + BUG_ON(block->header & DRM_BUDDY_HEADER_UNUSED); return block; } @@ -41,23 +45,49 @@ static void drm_block_free(struct drm_buddy *mm, kmem_cache_free(slab_blocks, block); } -static void list_insert_sorted(struct drm_buddy *mm, - struct drm_buddy_block *block) +static bool drm_buddy_block_offset_less(const struct drm_buddy_block *block, + const struct drm_buddy_block *node) { - struct drm_buddy_block *node; - struct list_head *head; + return drm_buddy_block_offset(block) < drm_buddy_block_offset(node); +} - head = &mm->free_list[drm_buddy_block_order(block)]; - if (list_empty(head)) { - list_add(&block->link, head); - return; - } +static bool rbtree_block_offset_less(struct rb_node *block, + const struct rb_node *node) +{ + return drm_buddy_block_offset_less(rbtree_get_free_block(block), + rbtree_get_free_block(node)); +} - list_for_each_entry(node, head, link) - if (drm_buddy_block_offset(block) < drm_buddy_block_offset(node)) - break; +static void rbtree_insert(struct drm_buddy *mm, + struct drm_buddy_block *block) +{ + rb_add(&block->rb, + &mm->free_tree[drm_buddy_block_order(block)], + rbtree_block_offset_less); +} + +static void rbtree_remove(struct drm_buddy *mm, + struct drm_buddy_block *block) +{ + struct rb_root *root; + + root = &mm->free_tree[drm_buddy_block_order(block)]; + rb_erase(&block->rb, root); - __list_add(&block->link, node->link.prev, &node->link); + RB_CLEAR_NODE(&block->rb); +} + +static struct drm_buddy_block * +rbtree_last_entry(struct drm_buddy *mm, unsigned int order) +{ + struct rb_node *node = rb_last(&mm->free_tree[order]); + + return node ? rb_entry(node, struct drm_buddy_block, rb) : NULL; +} + +static bool rbtree_is_empty(struct drm_buddy *mm, unsigned int order) +{ + return RB_EMPTY_ROOT(&mm->free_tree[order]); } static void clear_reset(struct drm_buddy_block *block) @@ -70,12 +100,13 @@ static void mark_cleared(struct drm_buddy_block *block) block->header |= DRM_BUDDY_HEADER_CLEAR; } -static void mark_allocated(struct drm_buddy_block *block) +static void mark_allocated(struct drm_buddy *mm, + struct drm_buddy_block *block) { block->header &= ~DRM_BUDDY_HEADER_STATE; block->header |= DRM_BUDDY_ALLOCATED; - list_del(&block->link); + rbtree_remove(mm, block); } static void mark_free(struct drm_buddy *mm, @@ -84,15 +115,16 @@ static void mark_free(struct drm_buddy *mm, block->header &= ~DRM_BUDDY_HEADER_STATE; block->header |= DRM_BUDDY_FREE; - list_insert_sorted(mm, block); + rbtree_insert(mm, block); } -static void mark_split(struct drm_buddy_block *block) +static void mark_split(struct drm_buddy *mm, + struct drm_buddy_block *block) { block->header &= ~DRM_BUDDY_HEADER_STATE; block->header |= DRM_BUDDY_SPLIT; - list_del(&block->link); + rbtree_remove(mm, block); } static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2) @@ -148,7 +180,7 @@ static unsigned int __drm_buddy_free(struct drm_buddy *mm, mark_cleared(parent); } - list_del(&buddy->link); + rbtree_remove(mm, buddy); if (force_merge && drm_buddy_block_is_clear(buddy)) mm->clear_avail -= drm_buddy_block_size(mm, buddy); @@ -179,9 +211,11 @@ static int __force_merge(struct drm_buddy *mm, return -EINVAL; for (i = min_order - 1; i >= 0; i--) { - struct drm_buddy_block *block, *prev; + struct drm_buddy_block *block, *prev_block, *first_block; + + first_block = rb_entry(rb_first(&mm->free_tree[i]), struct drm_buddy_block, rb); - list_for_each_entry_safe_reverse(block, prev, &mm->free_list[i], link) { + rbtree_reverse_for_each_entry_safe(block, prev_block, &mm->free_tree[i], rb) { struct drm_buddy_block *buddy; u64 block_start, block_end; @@ -206,10 +240,14 @@ static int __force_merge(struct drm_buddy *mm, * block in the next iteration as we would free the * buddy block as part of the free function. */ - if (prev == buddy) - prev = list_prev_entry(prev, link); + if (prev_block && prev_block == buddy) { + if (prev_block != first_block) + prev_block = rb_entry(rb_prev(&prev_block->rb), + struct drm_buddy_block, + rb); + } - list_del(&block->link); + rbtree_remove(mm, block); if (drm_buddy_block_is_clear(block)) mm->clear_avail -= drm_buddy_block_size(mm, block); @@ -258,14 +296,14 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) BUG_ON(mm->max_order > DRM_BUDDY_MAX_ORDER); - mm->free_list = kmalloc_array(mm->max_order + 1, - sizeof(struct list_head), + mm->free_tree = kmalloc_array(mm->max_order + 1, + sizeof(struct rb_root), GFP_KERNEL); - if (!mm->free_list) + if (!mm->free_tree) return -ENOMEM; for (i = 0; i <= mm->max_order; ++i) - INIT_LIST_HEAD(&mm->free_list[i]); + mm->free_tree[i] = RB_ROOT; mm->n_roots = hweight64(size); @@ -273,7 +311,7 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) sizeof(struct drm_buddy_block *), GFP_KERNEL); if (!mm->roots) - goto out_free_list; + goto out_free_tree; offset = 0; i = 0; @@ -312,8 +350,8 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) while (i--) drm_block_free(mm, mm->roots[i]); kfree(mm->roots); -out_free_list: - kfree(mm->free_list); +out_free_tree: + kfree(mm->free_tree); return -ENOMEM; } EXPORT_SYMBOL(drm_buddy_init); @@ -323,7 +361,7 @@ EXPORT_SYMBOL(drm_buddy_init); * * @mm: DRM buddy manager to free * - * Cleanup memory manager resources and the freelist + * Cleanup memory manager resources and the freetree */ void drm_buddy_fini(struct drm_buddy *mm) { @@ -350,7 +388,7 @@ void drm_buddy_fini(struct drm_buddy *mm) WARN_ON(mm->avail != mm->size); kfree(mm->roots); - kfree(mm->free_list); + kfree(mm->free_tree); } EXPORT_SYMBOL(drm_buddy_fini); @@ -383,7 +421,7 @@ static int split_block(struct drm_buddy *mm, clear_reset(block); } - mark_split(block); + mark_split(mm, block); return 0; } @@ -412,7 +450,7 @@ EXPORT_SYMBOL(drm_get_buddy); * @is_clear: blocks clear state * * Reset the clear state based on @is_clear value for each block - * in the freelist. + * in the freetree. */ void drm_buddy_reset_clear(struct drm_buddy *mm, bool is_clear) { @@ -433,7 +471,7 @@ void drm_buddy_reset_clear(struct drm_buddy *mm, bool is_clear) for (i = 0; i <= mm->max_order; ++i) { struct drm_buddy_block *block; - list_for_each_entry_reverse(block, &mm->free_list[i], link) { + rbtree_reverse_for_each_entry(block, &mm->free_tree[i], rb) { if (is_clear != drm_buddy_block_is_clear(block)) { if (is_clear) { mark_cleared(block); @@ -641,7 +679,7 @@ get_maxblock(struct drm_buddy *mm, unsigned int order, for (i = order; i <= mm->max_order; ++i) { struct drm_buddy_block *tmp_block; - list_for_each_entry_reverse(tmp_block, &mm->free_list[i], link) { + rbtree_reverse_for_each_entry(tmp_block, &mm->free_tree[i], rb) { if (block_incompatible(tmp_block, flags)) continue; @@ -667,7 +705,7 @@ get_maxblock(struct drm_buddy *mm, unsigned int order, } static struct drm_buddy_block * -alloc_from_freelist(struct drm_buddy *mm, +alloc_from_freetree(struct drm_buddy *mm, unsigned int order, unsigned long flags) { @@ -684,7 +722,7 @@ alloc_from_freelist(struct drm_buddy *mm, for (tmp = order; tmp <= mm->max_order; ++tmp) { struct drm_buddy_block *tmp_block; - list_for_each_entry_reverse(tmp_block, &mm->free_list[tmp], link) { + rbtree_reverse_for_each_entry(tmp_block, &mm->free_tree[tmp], rb) { if (block_incompatible(tmp_block, flags)) continue; @@ -700,10 +738,8 @@ alloc_from_freelist(struct drm_buddy *mm, if (!block) { /* Fallback method */ for (tmp = order; tmp <= mm->max_order; ++tmp) { - if (!list_empty(&mm->free_list[tmp])) { - block = list_last_entry(&mm->free_list[tmp], - struct drm_buddy_block, - link); + if (!rbtree_is_empty(mm, tmp)) { + block = rbtree_last_entry(mm, tmp); if (block) break; } @@ -771,7 +807,7 @@ static int __alloc_range(struct drm_buddy *mm, if (contains(start, end, block_start, block_end)) { if (drm_buddy_block_is_free(block)) { - mark_allocated(block); + mark_allocated(mm, block); total_allocated += drm_buddy_block_size(mm, block); mm->avail -= drm_buddy_block_size(mm, block); if (drm_buddy_block_is_clear(block)) @@ -849,7 +885,6 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm, { u64 rhs_offset, lhs_offset, lhs_size, filled; struct drm_buddy_block *block; - struct list_head *list; LIST_HEAD(blocks_lhs); unsigned long pages; unsigned int order; @@ -862,11 +897,10 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm, if (order == 0) return -ENOSPC; - list = &mm->free_list[order]; - if (list_empty(list)) + if (rbtree_is_empty(mm, order)) return -ENOSPC; - list_for_each_entry_reverse(block, list, link) { + rbtree_reverse_for_each_entry(block, &mm->free_tree[order], rb) { /* Allocate blocks traversing RHS */ rhs_offset = drm_buddy_block_offset(block); err = __drm_buddy_alloc_range(mm, rhs_offset, size, @@ -976,7 +1010,7 @@ int drm_buddy_block_trim(struct drm_buddy *mm, list_add(&block->tmp_link, &dfs); err = __alloc_range(mm, &dfs, new_start, new_size, blocks, NULL); if (err) { - mark_allocated(block); + mark_allocated(mm, block); mm->avail -= drm_buddy_block_size(mm, block); if (drm_buddy_block_is_clear(block)) mm->clear_avail -= drm_buddy_block_size(mm, block); @@ -999,8 +1033,8 @@ __drm_buddy_alloc_blocks(struct drm_buddy *mm, return __drm_buddy_alloc_range_bias(mm, start, end, order, flags); else - /* Allocate from freelist */ - return alloc_from_freelist(mm, order, flags); + /* Allocate from freetree */ + return alloc_from_freetree(mm, order, flags); } /** @@ -1017,8 +1051,8 @@ __drm_buddy_alloc_blocks(struct drm_buddy *mm, * alloc_range_bias() called on range limitations, which traverses * the tree and returns the desired block. * - * alloc_from_freelist() called when *no* range restrictions - * are enforced, which picks the block from the freelist. + * alloc_from_freetree() called when *no* range restrictions + * are enforced, which picks the block from the freetree. * * Returns: * 0 on success, error code on failure. @@ -1120,7 +1154,7 @@ int drm_buddy_alloc_blocks(struct drm_buddy *mm, } } while (1); - mark_allocated(block); + mark_allocated(mm, block); mm->avail -= drm_buddy_block_size(mm, block); if (drm_buddy_block_is_clear(block)) mm->clear_avail -= drm_buddy_block_size(mm, block); @@ -1204,7 +1238,7 @@ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p) struct drm_buddy_block *block; u64 count = 0, free; - list_for_each_entry(block, &mm->free_list[order], link) { + rbtree_for_each_entry(block, &mm->free_tree[order], rb) { BUG_ON(!drm_buddy_block_is_free(block)); count++; } diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h index 513837632b7d..9ee105d4309f 100644 --- a/include/drm/drm_buddy.h +++ b/include/drm/drm_buddy.h @@ -10,6 +10,7 @@ #include <linux/list.h> #include <linux/slab.h> #include <linux/sched.h> +#include <linux/rbtree.h> #include <drm/drm_print.h> @@ -53,7 +54,11 @@ struct drm_buddy_block { * a list, if so desired. As soon as the block is freed with * drm_buddy_free* ownership is given back to the mm. */ - struct list_head link; + union { + struct rb_node rb; + struct list_head link; + }; + struct list_head tmp_link; }; @@ -68,7 +73,7 @@ struct drm_buddy_block { */ struct drm_buddy { /* Maintain a free list for each order. */ - struct list_head *free_list; + struct rb_root *free_tree; /* * Maintain explicit binary tree(s) to track the allocation of the @@ -94,7 +99,7 @@ struct drm_buddy { }; static inline u64 -drm_buddy_block_offset(struct drm_buddy_block *block) +drm_buddy_block_offset(const struct drm_buddy_block *block) { return block->header & DRM_BUDDY_HEADER_OFFSET; } diff --git a/include/linux/rbtree.h b/include/linux/rbtree.h index 8d2ba3749866..17190bb4837c 100644 --- a/include/linux/rbtree.h +++ b/include/linux/rbtree.h @@ -79,6 +79,62 @@ static inline void rb_link_node_rcu(struct rb_node *node, struct rb_node *parent ____ptr ? rb_entry(____ptr, type, member) : NULL; \ }) +/** + * rbtree_for_each_entry - iterate in-order over rb_root of given type + * + * @pos: the 'type *' to use as a loop cursor. + * @root: 'rb_root *' of the rbtree. + * @member: the name of the rb_node field within 'type'. + */ +#define rbtree_for_each_entry(pos, root, member) \ + for ((pos) = rb_entry_safe(rb_first(root), typeof(*(pos)), member); \ + (pos); \ + (pos) = rb_entry_safe(rb_next(&(pos)->member), typeof(*(pos)), member)) + +/** + * rbtree_reverse_for_each_entry - iterate in reverse in-order over rb_root + * of given type + * + * @pos: the 'type *' to use as a loop cursor. + * @root: 'rb_root *' of the rbtree. + * @member: the name of the rb_node field within 'type'. + */ +#define rbtree_reverse_for_each_entry(pos, root, member) \ + for ((pos) = rb_entry_safe(rb_last(root), typeof(*(pos)), member); \ + (pos); \ + (pos) = rb_entry_safe(rb_prev(&(pos)->member), typeof(*(pos)), member)) + +/** + * rbtree_for_each_entry_safe - iterate in-order over rb_root safe against removal + * + * @pos: the 'type *' to use as a loop cursor + * @n: another 'type *' to use as temporary storage + * @root: 'rb_root *' of the rbtree + * @member: the name of the rb_node field within 'type' + */ +#define rbtree_for_each_entry_safe(pos, n, root, member) \ + for ((pos) = rb_entry_safe(rb_first(root), typeof(*(pos)), member), \ + (n) = (pos) ? rb_entry_safe(rb_next(&(pos)->member), typeof(*(pos)), member) : NULL; \ + (pos); \ + (pos) = (n), \ + (n) = (pos) ? rb_entry_safe(rb_next(&(pos)->member), typeof(*(pos)), member) : NULL) + +/** + * rbtree_reverse_for_each_entry_safe - iterate in reverse in-order over rb_root + * safe against removal + * + * @pos: the struct type * to use as a loop cursor. + * @n: another struct type * to use as temporary storage. + * @root: pointer to struct rb_root to iterate. + * @member: name of the rb_node field within the struct. + */ +#define rbtree_reverse_for_each_entry_safe(pos, n, root, member) \ + for ((pos) = rb_entry_safe(rb_last(root), typeof(*(pos)), member), \ + (n) = (pos) ? rb_entry_safe(rb_prev(&(pos)->member), typeof(*(pos)), member) : NULL; \ + (pos); \ + (pos) = (n), \ + (n) = (pos) ? rb_entry_safe(rb_prev(&(pos)->member), typeof(*(pos)), member) : NULL) + /** * rbtree_postorder_for_each_entry_safe - iterate in post-order over rb_root of * given type allowing the backing memory of @pos to be invalidated base-commit: 7156602d56e5ad689ae11e03680ab6326238b5e3 -- 2.34.1

5 days, 23 hours

1
1
0 0

[PATCH net v2] rds: ib: Increment i_fastreg_wrs before bailing out

by Håkon Bugge

We need to increment i_fastreg_wrs before we bail out from rds_ib_post_reg_frmr(). Fixes: 1659185fb4d0 ("RDS: IB: Support Fastreg MR (FRMR) memory registration mode") Fixes: 3a2886cca703 ("net/rds: Keep track of and wait for FRWR segments in use upon shutdown") Cc: stable(a)vger.kernel.org Signed-off-by: Håkon Bugge <haakon.bugge(a)oracle.com> --- v1 -> v2: Added Cc: stable(a)vger.kernel.org --- net/rds/ib_frmr.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/net/rds/ib_frmr.c b/net/rds/ib_frmr.c index 28c1b00221780..7e3b04a83904d 100644 --- a/net/rds/ib_frmr.c +++ b/net/rds/ib_frmr.c @@ -133,12 +133,15 @@ static int rds_ib_post_reg_frmr(struct rds_ib_mr *ibmr) ret = ib_map_mr_sg_zbva(frmr->mr, ibmr->sg, ibmr->sg_dma_len, &off, PAGE_SIZE); - if (unlikely(ret != ibmr->sg_dma_len)) - return ret < 0 ? ret : -EINVAL; + if (unlikely(ret != ibmr->sg_dma_len)) { + ret = ret < 0 ? ret : -EINVAL; + goto out_inc; + } - if (cmpxchg(&frmr->fr_state, - FRMR_IS_FREE, FRMR_IS_INUSE) != FRMR_IS_FREE) - return -EBUSY; + if (cmpxchg(&frmr->fr_state, FRMR_IS_FREE, FRMR_IS_INUSE) != FRMR_IS_FREE) { + ret = -EBUSY; + goto out_inc; + } atomic_inc(&ibmr->ic->i_fastreg_inuse_count); @@ -178,9 +181,11 @@ static int rds_ib_post_reg_frmr(struct rds_ib_mr *ibmr) * being accessed while registration is still pending. */ wait_event(frmr->fr_reg_done, !frmr->fr_reg); - out: + return ret; +out_inc: + atomic_inc(&ibmr->ic->i_fastreg_wrs); return ret; } -- 2.43.5

6 days

2
1
0 0

[PATCH v5 2/3] drm/buddy: Separate clear and dirty free block trees

by Arunpravin Paneer Selvam

Maintain two separate RB trees per order - one for clear (zeroed) blocks and another for dirty (uncleared) blocks. This separation improves code clarity and makes it more obvious which tree is being searched during allocation. It also improves scalability and efficiency when searching for a specific type of block, avoiding unnecessary checks and making the allocator more predictable under fragmentation. The changes have been validated using the existing drm_buddy_test KUnit test cases, along with selected graphics workloads, to ensure correctness and avoid regressions. v2: Missed adding the suggested-by tag. Added it in v2. v3(Matthew): - Remove the double underscores from the internal functions. - Rename the internal functions to have less generic names. - Fix the error handling code. - Pass tree argument for the tree macro. - Use the existing dirty/free bit instead of new tree field. - Make free_trees[] instead of clear_tree and dirty_tree for more cleaner approach. v4: - A bug was reported by Intel CI and it is fixed by Matthew Auld. - Replace the get_root function with &mm->free_trees[tree][order] (Matthew) - Remove the unnecessary rbtree_is_empty() check (Matthew) - Remove the unnecessary get_tree_for_flags() function. - Rename get_tree_for_block() name with get_block_tree() for more clarity. v5(Jani Nikula): - Don't use static inline in .c files. - enum free_tree and enumerator names are quite generic for a header and usage and the whole enum should be an implementation detail. Cc: stable(a)vger.kernel.org Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam(a)amd.com> Suggested-by: Matthew Auld <matthew.auld(a)intel.com> Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/4260 --- drivers/gpu/drm/drm_buddy.c | 336 +++++++++++++++++++++--------------- include/drm/drm_buddy.h | 2 +- 2 files changed, 201 insertions(+), 137 deletions(-) diff --git a/drivers/gpu/drm/drm_buddy.c b/drivers/gpu/drm/drm_buddy.c index 8b340f47f73d..4f96e2e17d08 100644 --- a/drivers/gpu/drm/drm_buddy.c +++ b/drivers/gpu/drm/drm_buddy.c @@ -12,8 +12,17 @@ #include <drm/drm_buddy.h> +enum drm_buddy_free_tree { + DRM_BUDDY_CLEAR_TREE = 0, + DRM_BUDDY_DIRTY_TREE, + DRM_BUDDY_MAX_FREE_TREES, +}; + static struct kmem_cache *slab_blocks; +#define for_each_free_tree(tree) \ + for ((tree) = 0; (tree) < DRM_BUDDY_MAX_FREE_TREES; (tree)++) + static struct drm_buddy_block *drm_block_alloc(struct drm_buddy *mm, struct drm_buddy_block *parent, unsigned int order, @@ -43,22 +52,61 @@ static void drm_block_free(struct drm_buddy *mm, kmem_cache_free(slab_blocks, block); } +static enum drm_buddy_free_tree +get_block_tree(struct drm_buddy_block *block) +{ + return drm_buddy_block_is_clear(block) ? + DRM_BUDDY_CLEAR_TREE : DRM_BUDDY_DIRTY_TREE; +} + +static struct drm_buddy_block * +rbtree_get_free_block(struct rb_node *node) +{ + return node ? rb_entry(node, struct drm_buddy_block, rb) : NULL; +} + +static struct drm_buddy_block * +rbtree_prev_free_block(struct rb_node *node) +{ + return rbtree_get_free_block(rb_prev(node)); +} + +static struct drm_buddy_block * +rbtree_first_free_block(struct rb_root *root) +{ + return rbtree_get_free_block(rb_first(root)); +} + +static struct drm_buddy_block * +rbtree_last_free_block(struct rb_root *root) +{ + return rbtree_get_free_block(rb_last(root)); +} + +static bool rbtree_is_empty(struct rb_root *root) +{ + return RB_EMPTY_ROOT(root); +} + static void rbtree_insert(struct drm_buddy *mm, - struct drm_buddy_block *block) + struct drm_buddy_block *block, + enum drm_buddy_free_tree tree) { - struct rb_root *root = &mm->free_tree[drm_buddy_block_order(block)]; - struct rb_node **link = &root->rb_node; - struct rb_node *parent = NULL; + struct rb_node **link, *parent = NULL; struct drm_buddy_block *node; - u64 offset; + struct rb_root *root; + unsigned int order; - offset = drm_buddy_block_offset(block); + order = drm_buddy_block_order(block); + + root = &mm->free_trees[tree][order]; + link = &root->rb_node; while (*link) { parent = *link; - node = rb_entry(parent, struct drm_buddy_block, rb); + node = rbtree_get_free_block(parent); - if (offset < drm_buddy_block_offset(node)) + if (drm_buddy_block_offset(block) < drm_buddy_block_offset(node)) link = &parent->rb_left; else link = &parent->rb_right; @@ -71,27 +119,17 @@ static void rbtree_insert(struct drm_buddy *mm, static void rbtree_remove(struct drm_buddy *mm, struct drm_buddy_block *block) { + unsigned int order = drm_buddy_block_order(block); struct rb_root *root; + enum drm_buddy_free_tree tree; - root = &mm->free_tree[drm_buddy_block_order(block)]; - rb_erase(&block->rb, root); + tree = get_block_tree(block); + root = &mm->free_trees[tree][order]; + rb_erase(&block->rb, root); RB_CLEAR_NODE(&block->rb); } -static struct drm_buddy_block * -rbtree_last_entry(struct drm_buddy *mm, unsigned int order) -{ - struct rb_node *node = rb_last(&mm->free_tree[order]); - - return node ? rb_entry(node, struct drm_buddy_block, rb) : NULL; -} - -static bool rbtree_is_empty(struct drm_buddy *mm, unsigned int order) -{ - return RB_EMPTY_ROOT(&mm->free_tree[order]); -} - static void clear_reset(struct drm_buddy_block *block) { block->header &= ~DRM_BUDDY_HEADER_CLEAR; @@ -114,10 +152,13 @@ static void mark_allocated(struct drm_buddy *mm, static void mark_free(struct drm_buddy *mm, struct drm_buddy_block *block) { + enum drm_buddy_free_tree tree; + block->header &= ~DRM_BUDDY_HEADER_STATE; block->header |= DRM_BUDDY_FREE; - rbtree_insert(mm, block); + tree = get_block_tree(block); + rbtree_insert(mm, block, tree); } static void mark_split(struct drm_buddy *mm, @@ -203,6 +244,7 @@ static int __force_merge(struct drm_buddy *mm, u64 end, unsigned int min_order) { + enum drm_buddy_free_tree tree; unsigned int order; int i; @@ -212,50 +254,49 @@ static int __force_merge(struct drm_buddy *mm, if (min_order > mm->max_order) return -EINVAL; - for (i = min_order - 1; i >= 0; i--) { - struct drm_buddy_block *block, *prev_block, *first_block; - - first_block = rb_entry(rb_first(&mm->free_tree[i]), struct drm_buddy_block, rb); + for_each_free_tree(tree) { + for (i = min_order - 1; i >= 0; i--) { + struct rb_root *root = &mm->free_trees[tree][i]; + struct drm_buddy_block *block, *prev_block; - rbtree_reverse_for_each_entry_safe(block, prev_block, &mm->free_tree[i], rb) { - struct drm_buddy_block *buddy; - u64 block_start, block_end; + rbtree_reverse_for_each_entry_safe(block, prev_block, root, rb) { + struct drm_buddy_block *buddy; + u64 block_start, block_end; - if (!block->parent) - continue; + if (!block->parent) + continue; - block_start = drm_buddy_block_offset(block); - block_end = block_start + drm_buddy_block_size(mm, block) - 1; + block_start = drm_buddy_block_offset(block); + block_end = block_start + drm_buddy_block_size(mm, block) - 1; - if (!contains(start, end, block_start, block_end)) - continue; + if (!contains(start, end, block_start, block_end)) + continue; - buddy = __get_buddy(block); - if (!drm_buddy_block_is_free(buddy)) - continue; + buddy = __get_buddy(block); + if (!drm_buddy_block_is_free(buddy)) + continue; - WARN_ON(drm_buddy_block_is_clear(block) == - drm_buddy_block_is_clear(buddy)); + WARN_ON(drm_buddy_block_is_clear(block) == + drm_buddy_block_is_clear(buddy)); - /* - * If the prev block is same as buddy, don't access the - * block in the next iteration as we would free the - * buddy block as part of the free function. - */ - if (prev_block && prev_block == buddy) { - if (prev_block != first_block) - prev_block = rb_entry(rb_prev(&prev_block->rb), - struct drm_buddy_block, - rb); - } + /* + * If the prev block is same as buddy, don't access the + * block in the next iteration as we would free the + * buddy block as part of the free function. + */ + if (prev_block && prev_block == buddy) { + if (prev_block != rbtree_first_free_block(root)) + prev_block = rbtree_prev_free_block(&prev_block->rb); + } - rbtree_remove(mm, block); - if (drm_buddy_block_is_clear(block)) - mm->clear_avail -= drm_buddy_block_size(mm, block); + rbtree_remove(mm, block); + if (drm_buddy_block_is_clear(block)) + mm->clear_avail -= drm_buddy_block_size(mm, block); - order = __drm_buddy_free(mm, block, true); - if (order >= min_order) - return 0; + order = __drm_buddy_free(mm, block, true); + if (order >= min_order) + return 0; + } } } @@ -276,7 +317,7 @@ static int __force_merge(struct drm_buddy *mm, */ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) { - unsigned int i; + unsigned int i, j; u64 offset; if (size < chunk_size) @@ -298,14 +339,22 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) BUG_ON(mm->max_order > DRM_BUDDY_MAX_ORDER); - mm->free_tree = kmalloc_array(mm->max_order + 1, - sizeof(struct rb_root), - GFP_KERNEL); - if (!mm->free_tree) + mm->free_trees = kmalloc_array(DRM_BUDDY_MAX_FREE_TREES, + sizeof(*mm->free_trees), + GFP_KERNEL); + if (!mm->free_trees) return -ENOMEM; - for (i = 0; i <= mm->max_order; ++i) - mm->free_tree[i] = RB_ROOT; + for (i = 0; i < DRM_BUDDY_MAX_FREE_TREES; i++) { + mm->free_trees[i] = kmalloc_array(mm->max_order + 1, + sizeof(struct rb_root), + GFP_KERNEL); + if (!mm->free_trees[i]) + goto out_free_tree; + + for (j = 0; j <= mm->max_order; ++j) + mm->free_trees[i][j] = RB_ROOT; + } mm->n_roots = hweight64(size); @@ -353,7 +402,9 @@ int drm_buddy_init(struct drm_buddy *mm, u64 size, u64 chunk_size) drm_block_free(mm, mm->roots[i]); kfree(mm->roots); out_free_tree: - kfree(mm->free_tree); + while (i--) + kfree(mm->free_trees[i]); + kfree(mm->free_trees); return -ENOMEM; } EXPORT_SYMBOL(drm_buddy_init); @@ -389,8 +440,9 @@ void drm_buddy_fini(struct drm_buddy *mm) WARN_ON(mm->avail != mm->size); + for (i = 0; i < DRM_BUDDY_MAX_FREE_TREES; i++) + kfree(mm->free_trees[i]); kfree(mm->roots); - kfree(mm->free_tree); } EXPORT_SYMBOL(drm_buddy_fini); @@ -414,8 +466,7 @@ static int split_block(struct drm_buddy *mm, return -ENOMEM; } - mark_free(mm, block->left); - mark_free(mm, block->right); + mark_split(mm, block); if (drm_buddy_block_is_clear(block)) { mark_cleared(block->left); @@ -423,7 +474,8 @@ static int split_block(struct drm_buddy *mm, clear_reset(block); } - mark_split(mm, block); + mark_free(mm, block->left); + mark_free(mm, block->right); return 0; } @@ -456,6 +508,7 @@ EXPORT_SYMBOL(drm_get_buddy); */ void drm_buddy_reset_clear(struct drm_buddy *mm, bool is_clear) { + enum drm_buddy_free_tree src_tree, dst_tree; u64 root_size, size, start; unsigned int order; int i; @@ -470,19 +523,24 @@ void drm_buddy_reset_clear(struct drm_buddy *mm, bool is_clear) size -= root_size; } + src_tree = is_clear ? DRM_BUDDY_DIRTY_TREE : DRM_BUDDY_CLEAR_TREE; + dst_tree = is_clear ? DRM_BUDDY_CLEAR_TREE : DRM_BUDDY_DIRTY_TREE; + for (i = 0; i <= mm->max_order; ++i) { + struct rb_root *root = &mm->free_trees[src_tree][i]; struct drm_buddy_block *block; - rbtree_reverse_for_each_entry(block, &mm->free_tree[i], rb) { - if (is_clear != drm_buddy_block_is_clear(block)) { - if (is_clear) { - mark_cleared(block); - mm->clear_avail += drm_buddy_block_size(mm, block); - } else { - clear_reset(block); - mm->clear_avail -= drm_buddy_block_size(mm, block); - } + rbtree_reverse_for_each_entry(block, root, rb) { + rbtree_remove(mm, block); + if (is_clear) { + mark_cleared(block); + mm->clear_avail += drm_buddy_block_size(mm, block); + } else { + clear_reset(block); + mm->clear_avail -= drm_buddy_block_size(mm, block); } + + rbtree_insert(mm, block, dst_tree); } } } @@ -672,23 +730,17 @@ __drm_buddy_alloc_range_bias(struct drm_buddy *mm, } static struct drm_buddy_block * -get_maxblock(struct drm_buddy *mm, unsigned int order, - unsigned long flags) +get_maxblock(struct drm_buddy *mm, + unsigned int order, + enum drm_buddy_free_tree tree) { struct drm_buddy_block *max_block = NULL, *block = NULL; + struct rb_root *root; unsigned int i; for (i = order; i <= mm->max_order; ++i) { - struct drm_buddy_block *tmp_block; - - rbtree_reverse_for_each_entry(tmp_block, &mm->free_tree[i], rb) { - if (block_incompatible(tmp_block, flags)) - continue; - - block = tmp_block; - break; - } - + root = &mm->free_trees[tree][i]; + block = rbtree_last_free_block(root); if (!block) continue; @@ -712,39 +764,39 @@ alloc_from_freetree(struct drm_buddy *mm, unsigned long flags) { struct drm_buddy_block *block = NULL; + struct rb_root *root; + enum drm_buddy_free_tree tree; unsigned int tmp; int err; + tree = (flags & DRM_BUDDY_CLEAR_ALLOCATION) ? + DRM_BUDDY_CLEAR_TREE : DRM_BUDDY_DIRTY_TREE; + if (flags & DRM_BUDDY_TOPDOWN_ALLOCATION) { - block = get_maxblock(mm, order, flags); + block = get_maxblock(mm, order, tree); if (block) /* Store the obtained block order */ tmp = drm_buddy_block_order(block); } else { for (tmp = order; tmp <= mm->max_order; ++tmp) { - struct drm_buddy_block *tmp_block; - - rbtree_reverse_for_each_entry(tmp_block, &mm->free_tree[tmp], rb) { - if (block_incompatible(tmp_block, flags)) - continue; - - block = tmp_block; - break; - } - + /* Get RB tree root for this order and tree */ + root = &mm->free_trees[tree][tmp]; + block = rbtree_last_free_block(root); if (block) break; } } if (!block) { - /* Fallback method */ + /* Try allocating from the other tree */ + tree = (tree == DRM_BUDDY_CLEAR_TREE) ? + DRM_BUDDY_DIRTY_TREE : DRM_BUDDY_CLEAR_TREE; + for (tmp = order; tmp <= mm->max_order; ++tmp) { - if (!rbtree_is_empty(mm, tmp)) { - block = rbtree_last_entry(mm, tmp); - if (block) - break; - } + root = &mm->free_trees[tree][tmp]; + block = rbtree_last_free_block(root); + if (block) + break; } if (!block) @@ -888,6 +940,7 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm, u64 rhs_offset, lhs_offset, lhs_size, filled; struct drm_buddy_block *block; LIST_HEAD(blocks_lhs); + enum drm_buddy_free_tree tree; unsigned long pages; unsigned int order; u64 modify_size; @@ -899,34 +952,39 @@ static int __alloc_contig_try_harder(struct drm_buddy *mm, if (order == 0) return -ENOSPC; - if (rbtree_is_empty(mm, order)) + if (rbtree_is_empty(&mm->free_trees[DRM_BUDDY_CLEAR_TREE][order]) && + rbtree_is_empty(&mm->free_trees[DRM_BUDDY_DIRTY_TREE][order])) return -ENOSPC; - rbtree_reverse_for_each_entry(block, &mm->free_tree[order], rb) { - /* Allocate blocks traversing RHS */ - rhs_offset = drm_buddy_block_offset(block); - err = __drm_buddy_alloc_range(mm, rhs_offset, size, - &filled, blocks); - if (!err || err != -ENOSPC) - return err; - - lhs_size = max((size - filled), min_block_size); - if (!IS_ALIGNED(lhs_size, min_block_size)) - lhs_size = round_up(lhs_size, min_block_size); - - /* Allocate blocks traversing LHS */ - lhs_offset = drm_buddy_block_offset(block) - lhs_size; - err = __drm_buddy_alloc_range(mm, lhs_offset, lhs_size, - NULL, &blocks_lhs); - if (!err) { - list_splice(&blocks_lhs, blocks); - return 0; - } else if (err != -ENOSPC) { + for_each_free_tree(tree) { + struct rb_root *root = &mm->free_trees[tree][order]; + + rbtree_reverse_for_each_entry(block, root, rb) { + /* Allocate blocks traversing RHS */ + rhs_offset = drm_buddy_block_offset(block); + err = __drm_buddy_alloc_range(mm, rhs_offset, size, + &filled, blocks); + if (!err || err != -ENOSPC) + return err; + + lhs_size = max((size - filled), min_block_size); + if (!IS_ALIGNED(lhs_size, min_block_size)) + lhs_size = round_up(lhs_size, min_block_size); + + /* Allocate blocks traversing LHS */ + lhs_offset = drm_buddy_block_offset(block) - lhs_size; + err = __drm_buddy_alloc_range(mm, lhs_offset, lhs_size, + NULL, &blocks_lhs); + if (!err) { + list_splice(&blocks_lhs, blocks); + return 0; + } else if (err != -ENOSPC) { + drm_buddy_free_list_internal(mm, blocks); + return err; + } + /* Free blocks for the next iteration */ drm_buddy_free_list_internal(mm, blocks); - return err; } - /* Free blocks for the next iteration */ - drm_buddy_free_list_internal(mm, blocks); } return -ENOSPC; @@ -1231,6 +1289,7 @@ EXPORT_SYMBOL(drm_buddy_block_print); */ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p) { + enum drm_buddy_free_tree tree; int order; drm_printf(p, "chunk_size: %lluKiB, total: %lluMiB, free: %lluMiB, clear_free: %lluMiB\n", @@ -1238,11 +1297,16 @@ void drm_buddy_print(struct drm_buddy *mm, struct drm_printer *p) for (order = mm->max_order; order >= 0; order--) { struct drm_buddy_block *block; + struct rb_root *root; u64 count = 0, free; - rbtree_for_each_entry(block, &mm->free_tree[order], rb) { - BUG_ON(!drm_buddy_block_is_free(block)); - count++; + for_each_free_tree(tree) { + root = &mm->free_trees[tree][order]; + + rbtree_for_each_entry(block, root, rb) { + BUG_ON(!drm_buddy_block_is_free(block)); + count++; + } } drm_printf(p, "order-%2d ", order); diff --git a/include/drm/drm_buddy.h b/include/drm/drm_buddy.h index 091823592034..a2189a92bb7a 100644 --- a/include/drm/drm_buddy.h +++ b/include/drm/drm_buddy.h @@ -73,7 +73,7 @@ struct drm_buddy_block { */ struct drm_buddy { /* Maintain a free list for each order. */ - struct rb_root *free_tree; + struct rb_root **free_trees; /* * Maintain explicit binary tree(s) to track the allocation of the -- 2.34.1

6 days, 1 hour

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-stable-mirror September 2025