October 2022 - Linux-stable-mirror

FAILED: patch "[PATCH] mm/migrate_device.c: add missing flush_cache_page()" failed to apply to 5.10-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 5.10-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: a3589e1d5fe3 ("mm/migrate_device.c: add missing flush_cache_page()") 6c287605fd56 ("mm: remember exclusively mapped anonymous pages with PG_anon_exclusive") 6c54dc6c7437 ("mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively") 28c5209dfd5f ("mm/rmap: pass rmap flags to hugepage_add_anon_rmap()") f1e2db12e45b ("mm/rmap: remove do_page_add_anon_rmap()") 14f9135d5470 ("mm/rmap: convert RMAP flags to a proper distinct rmap_t type") fb3d824d1a46 ("mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and page_try_dup_anon_rmap()") b51ad4f8679e ("mm/memory: slightly simplify copy_present_pte()") 623a1ddfeb23 ("mm/hugetlb: take src_mm->write_protect_seq in copy_hugetlb_page_range()") 3bff7e3f1f16 ("mm/huge_memory: streamline COW logic in do_huge_pmd_wp_page()") c145e0b47c77 ("mm: streamline COW logic in do_swap_page()") 84d60fdd3733 ("mm: slightly clarify KSM logic in do_swap_page()") 53a05ad9f21d ("mm: optimize do_wp_page() for exclusive pages in the swapcache") 9030fb0bb9d6 ("Merge tag 'folio-5.18c' of git://git.infradead.org/users/willy/pagecache") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From a3589e1d5fe39c3d9fdd291b111524b93d08bc32 Mon Sep 17 00:00:00 2001 From: Alistair Popple <apopple(a)nvidia.com> Date: Fri, 2 Sep 2022 10:35:52 +1000 Subject: [PATCH] mm/migrate_device.c: add missing flush_cache_page() Currently we only call flush_cache_page() for the anon_exclusive case, however in both cases we clear the pte so should flush the cache. Link: https://lkml.kernel.org/r/5676f30436ab71d1a587ac73f835ed8bd2113ff5.16620785… Fixes: 8c3328f1f36a ("mm/migrate: migrate_vma() unmap page from vma while collecting pages") Signed-off-by: Alistair Popple <apopple(a)nvidia.com> Reviewed-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Cc: Alex Sierra <alex.sierra(a)amd.com> Cc: Ben Skeggs <bskeggs(a)redhat.com> Cc: Felix Kuehling <Felix.Kuehling(a)amd.com> Cc: huang ying <huang.ying.caritas(a)gmail.com> Cc: "Huang, Ying" <ying.huang(a)intel.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: John Hubbard <jhubbard(a)nvidia.com> Cc: Karol Herbst <kherbst(a)redhat.com> Cc: Logan Gunthorpe <logang(a)deltatee.com> Cc: Lyude Paul <lyude(a)redhat.com> Cc: Matthew Wilcox <willy(a)infradead.org> Cc: Nadav Amit <nadav.amit(a)gmail.com> Cc: Paul Mackerras <paulus(a)ozlabs.org> Cc: Ralph Campbell <rcampbell(a)nvidia.com> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6a5ef9f0da6a..4cc849c3b54c 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -193,9 +193,9 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, bool anon_exclusive; pte_t swp_pte; + flush_cache_page(vma, addr, pte_pfn(*ptep)); anon_exclusive = PageAnon(page) && PageAnonExclusive(page); if (anon_exclusive) { - flush_cache_page(vma, addr, pte_pfn(*ptep)); ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) {

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm/migrate_device.c: add missing flush_cache_page()" failed to apply to 5.15-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 5.15-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: a3589e1d5fe3 ("mm/migrate_device.c: add missing flush_cache_page()") 6c287605fd56 ("mm: remember exclusively mapped anonymous pages with PG_anon_exclusive") 6c54dc6c7437 ("mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively") 28c5209dfd5f ("mm/rmap: pass rmap flags to hugepage_add_anon_rmap()") f1e2db12e45b ("mm/rmap: remove do_page_add_anon_rmap()") 14f9135d5470 ("mm/rmap: convert RMAP flags to a proper distinct rmap_t type") fb3d824d1a46 ("mm/rmap: split page_dup_rmap() into page_dup_file_rmap() and page_try_dup_anon_rmap()") b51ad4f8679e ("mm/memory: slightly simplify copy_present_pte()") 623a1ddfeb23 ("mm/hugetlb: take src_mm->write_protect_seq in copy_hugetlb_page_range()") 3bff7e3f1f16 ("mm/huge_memory: streamline COW logic in do_huge_pmd_wp_page()") c145e0b47c77 ("mm: streamline COW logic in do_swap_page()") 84d60fdd3733 ("mm: slightly clarify KSM logic in do_swap_page()") 53a05ad9f21d ("mm: optimize do_wp_page() for exclusive pages in the swapcache") 9030fb0bb9d6 ("Merge tag 'folio-5.18c' of git://git.infradead.org/users/willy/pagecache") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From a3589e1d5fe39c3d9fdd291b111524b93d08bc32 Mon Sep 17 00:00:00 2001 From: Alistair Popple <apopple(a)nvidia.com> Date: Fri, 2 Sep 2022 10:35:52 +1000 Subject: [PATCH] mm/migrate_device.c: add missing flush_cache_page() Currently we only call flush_cache_page() for the anon_exclusive case, however in both cases we clear the pte so should flush the cache. Link: https://lkml.kernel.org/r/5676f30436ab71d1a587ac73f835ed8bd2113ff5.16620785… Fixes: 8c3328f1f36a ("mm/migrate: migrate_vma() unmap page from vma while collecting pages") Signed-off-by: Alistair Popple <apopple(a)nvidia.com> Reviewed-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Cc: Alex Sierra <alex.sierra(a)amd.com> Cc: Ben Skeggs <bskeggs(a)redhat.com> Cc: Felix Kuehling <Felix.Kuehling(a)amd.com> Cc: huang ying <huang.ying.caritas(a)gmail.com> Cc: "Huang, Ying" <ying.huang(a)intel.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: John Hubbard <jhubbard(a)nvidia.com> Cc: Karol Herbst <kherbst(a)redhat.com> Cc: Logan Gunthorpe <logang(a)deltatee.com> Cc: Lyude Paul <lyude(a)redhat.com> Cc: Matthew Wilcox <willy(a)infradead.org> Cc: Nadav Amit <nadav.amit(a)gmail.com> Cc: Paul Mackerras <paulus(a)ozlabs.org> Cc: Ralph Campbell <rcampbell(a)nvidia.com> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6a5ef9f0da6a..4cc849c3b54c 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -193,9 +193,9 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, bool anon_exclusive; pte_t swp_pte; + flush_cache_page(vma, addr, pte_pfn(*ptep)); anon_exclusive = PageAnon(page) && PageAnonExclusive(page); if (anon_exclusive) { - flush_cache_page(vma, addr, pte_pfn(*ptep)); ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) {

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 4.9-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 4.9-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas") 43ca106fa8ec ("mm: cma: support sysfs") bbb269206f3c ("mm: vmstat: add cma statistics") 1fbaf8fc12a0 ("mm: add a io_mapping_map_user helper") ac73e3dc8acd ("Merge branch 'akpm' (patches from Andrew)") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 4.14-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 4.14-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas") 43ca106fa8ec ("mm: cma: support sysfs") bbb269206f3c ("mm: vmstat: add cma statistics") 1fbaf8fc12a0 ("mm: add a io_mapping_map_user helper") ac73e3dc8acd ("Merge branch 'akpm' (patches from Andrew)") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 4.19-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 4.19-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas") 43ca106fa8ec ("mm: cma: support sysfs") bbb269206f3c ("mm: vmstat: add cma statistics") 1fbaf8fc12a0 ("mm: add a io_mapping_map_user helper") ac73e3dc8acd ("Merge branch 'akpm' (patches from Andrew)") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 5.4-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 5.4-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas") 43ca106fa8ec ("mm: cma: support sysfs") bbb269206f3c ("mm: vmstat: add cma statistics") 1fbaf8fc12a0 ("mm: add a io_mapping_map_user helper") ac73e3dc8acd ("Merge branch 'akpm' (patches from Andrew)") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 5.10-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 5.10-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") 1507f51255c9 ("mm: introduce memfd_secret system call to create "secret" memory areas") 43ca106fa8ec ("mm: cma: support sysfs") bbb269206f3c ("mm: vmstat: add cma statistics") 1fbaf8fc12a0 ("mm: add a io_mapping_map_user helper") ac73e3dc8acd ("Merge branch 'akpm' (patches from Andrew)") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mm: gup: fix the fast GUP race against THP collapse" failed to apply to 5.15-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 5.15-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: 70cbc3cc78a9 ("mm: gup: fix the fast GUP race against THP collapse") b0496fe4effd ("mm/gup: Convert gup_pte_range() to use a folio") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From 70cbc3cc78a997d8247b50389d37c4e1736019da Mon Sep 17 00:00:00 2001 From: Yang Shi <shy828301(a)gmail.com> Date: Wed, 7 Sep 2022 11:01:43 -0700 Subject: [PATCH] mm: gup: fix the fast GUP race against THP collapse Since general RCU GUP fast was introduced in commit 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer sufficient to handle concurrent GUP-fast in all cases, it only handles traditional IPI-based GUP-fast correctly. On architectures that send an IPI broadcast on TLB flush, it works as expected. But on the architectures that do not use IPI to broadcast TLB flush, it may have the below race: CPU A CPU B THP collapse fast GUP gup_pmd_range() <-- see valid pmd gup_pte_range() <-- work on pte pmdp_collapse_flush() <-- clear pmd and flush __collapse_huge_page_isolate() check page pinned <-- before GUP bump refcount pin the page check PTE <-- no change __collapse_huge_page_copy() copy data to huge page ptep_clear() install huge pmd for the huge page return the stale page discard the stale page The race can be fixed by checking whether PMD is changed or not after taking the page pin in fast GUP, just like what it does for PTE. If the PMD is changed it means there may be parallel THP collapse, so GUP should back off. Also update the stale comment about serializing against fast GUP in khugepaged. Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com Fixes: 2667f50e8b81 ("mm: introduce a general RCU get_user_pages_fast()") Acked-by: David Hildenbrand <david(a)redhat.com> Acked-by: Peter Xu <peterx(a)redhat.com> Signed-off-by: Yang Shi <shy828301(a)gmail.com> Reviewed-by: John Hubbard <jhubbard(a)nvidia.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar(a)linux.ibm.com> Cc: Hugh Dickins <hughd(a)google.com> Cc: Jason Gunthorpe <jgg(a)nvidia.com> Cc: "Kirill A. Shutemov" <kirill.shutemov(a)linux.intel.com> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Nicholas Piggin <npiggin(a)gmail.com> Cc: Christophe Leroy <christophe.leroy(a)csgroup.eu> Cc: <stable(a)vger.kernel.org> Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org> diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..00926abb4426 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2345,8 +2345,28 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +/* + * Fast-gup relies on pte change detection to avoid concurrent pgtable + * operations. + * + * To pin the page, fast-gup needs to do below in order: + * (1) pin the page (by prefetching pte), then (2) check pte not changed. + * + * For the rest of pgtable operations where pgtable updates can be racy + * with fast-gup, we need to do (1) clear pte, then (2) check whether page + * is pinned. + * + * Above will work for all pte-level operations, including THP split. + * + * For THP collapse, it's a bit more complicated because fast-gup may be + * walking a pgtable page that is being freed (pte is still valid but pmd + * can be cleared already). To avoid race in such condition, we need to + * also check pmd here to make sure pmd doesn't change (corresponds to + * pmdp_collapse_flush() in the THP collapse code path). + */ +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { struct dev_pagemap *pgmap = NULL; int nr_start = *nr, ret = 0; @@ -2392,7 +2412,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap; } - if (unlikely(pte_val(pte) != pte_val(*ptep))) { + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || + unlikely(pte_val(pte) != pte_val(*ptep))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2439,8 +2460,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, * get_user_pages_fast_only implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, - unsigned int flags, struct page **pages, int *nr) +static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) { return 0; } @@ -2764,7 +2786,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + } else if (!gup_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; } while (pmdp++, addr = next, addr != end); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..70b7ac66411c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1083,10 +1083,12 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* - * After this gup_fast can't run anymore. This also removes - * any huge TLB entry from the CPU so we won't allow - * huge and small TLB entries for the same virtual address - * to avoid the risk of CPU bugs in that area. + * This removes any huge TLB entry from the CPU so we won't allow + * huge and small TLB entries for the same virtual address to + * avoid the risk of CPU bugs in that area. + * + * Parallel fast GUP is fine since fast GUP will back off when + * it detects PMD is changed. */ _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl);

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mmc: core: Terminate infinite loop in SD-UHS voltage switch" failed to apply to 4.9-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 4.9-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: e9233917a7e5 ("mmc: core: Terminate infinite loop in SD-UHS voltage switch") e42726646082 ("mmc: core: Replace with already defined values for readability") 09247e110b2e ("mmc: core: Allow UHS-I voltage switch for SDSC cards if supported") 2ed573b603f7 ("mmc: core: Clarify usage of mmc_set_signal_voltage()") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From e9233917a7e53980664efbc565888163c0a33c3f Mon Sep 17 00:00:00 2001 From: Brian Norris <briannorris(a)chromium.org> Date: Tue, 13 Sep 2022 18:40:10 -0700 Subject: [PATCH] mmc: core: Terminate infinite loop in SD-UHS voltage switch This loop intends to retry a max of 10 times, with some implicit termination based on the SD_{R,}OCR_S18A bit. Unfortunately, the termination condition depends on the value reported by the SD card (*rocr), which may or may not correctly reflect what we asked it to do. Needless to say, it's not wise to rely on the card doing what we expect; we should at least terminate the loop regardless. So, check both the input and output values, so we ensure we will terminate regardless of the SD card behavior. Note that SDIO learned a similar retry loop in commit 0797e5f1453b ("mmc: core: Fixup signal voltage switch"), but that used the 'ocr' result, and so the current pre-terminating condition looks like: rocr & ocr & R4_18V_PRESENT (i.e., it doesn't have the same bug.) This addresses a number of crash reports seen on ChromeOS that look like the following: ... // lots of repeated: ... <4>[13142.846061] mmc1: Skipping voltage switch <4>[13143.406087] mmc1: Skipping voltage switch <4>[13143.964724] mmc1: Skipping voltage switch <4>[13144.526089] mmc1: Skipping voltage switch <4>[13145.086088] mmc1: Skipping voltage switch <4>[13145.645941] mmc1: Skipping voltage switch <3>[13146.153969] INFO: task halt:30352 blocked for more than 122 seconds. ... Fixes: f2119df6b764 ("mmc: sd: add support for signal voltage switch procedure") Cc: <stable(a)vger.kernel.org> Signed-off-by: Brian Norris <briannorris(a)chromium.org> Reviewed-by: Guenter Roeck <linux(a)roeck-us.net> Link: https://lore.kernel.org/r/20220914014010.2076169-1-briannorris@chromium.org Signed-off-by: Ulf Hansson <ulf.hansson(a)linaro.org> diff --git a/drivers/mmc/core/sd.c b/drivers/mmc/core/sd.c index 06aa62ce0ed1..3662bf5320ce 100644 --- a/drivers/mmc/core/sd.c +++ b/drivers/mmc/core/sd.c @@ -870,7 +870,8 @@ int mmc_sd_get_cid(struct mmc_host *host, u32 ocr, u32 *cid, u32 *rocr) * the CCS bit is set as well. We deliberately deviate from the spec in * regards to this, which allows UHS-I to be supported for SDSC cards. */ - if (!mmc_host_is_spi(host) && rocr && (*rocr & SD_ROCR_S18A)) { + if (!mmc_host_is_spi(host) && (ocr & SD_OCR_S18R) && + rocr && (*rocr & SD_ROCR_S18A)) { err = mmc_set_uhs_voltage(host, pocr); if (err == -EAGAIN) { retries--;

2 years, 2 months

1
0
0 0

FAILED: patch "[PATCH] mmc: core: Terminate infinite loop in SD-UHS voltage switch" failed to apply to 4.14-stable tree

by gregkh＠linuxfoundation.org

The patch below does not apply to the 4.14-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <stable(a)vger.kernel.org>. Possible dependencies: e9233917a7e5 ("mmc: core: Terminate infinite loop in SD-UHS voltage switch") e42726646082 ("mmc: core: Replace with already defined values for readability") 09247e110b2e ("mmc: core: Allow UHS-I voltage switch for SDSC cards if supported") thanks, greg k-h ------------------ original commit in Linus's tree ------------------ From e9233917a7e53980664efbc565888163c0a33c3f Mon Sep 17 00:00:00 2001 From: Brian Norris <briannorris(a)chromium.org> Date: Tue, 13 Sep 2022 18:40:10 -0700 Subject: [PATCH] mmc: core: Terminate infinite loop in SD-UHS voltage switch This loop intends to retry a max of 10 times, with some implicit termination based on the SD_{R,}OCR_S18A bit. Unfortunately, the termination condition depends on the value reported by the SD card (*rocr), which may or may not correctly reflect what we asked it to do. Needless to say, it's not wise to rely on the card doing what we expect; we should at least terminate the loop regardless. So, check both the input and output values, so we ensure we will terminate regardless of the SD card behavior. Note that SDIO learned a similar retry loop in commit 0797e5f1453b ("mmc: core: Fixup signal voltage switch"), but that used the 'ocr' result, and so the current pre-terminating condition looks like: rocr & ocr & R4_18V_PRESENT (i.e., it doesn't have the same bug.) This addresses a number of crash reports seen on ChromeOS that look like the following: ... // lots of repeated: ... <4>[13142.846061] mmc1: Skipping voltage switch <4>[13143.406087] mmc1: Skipping voltage switch <4>[13143.964724] mmc1: Skipping voltage switch <4>[13144.526089] mmc1: Skipping voltage switch <4>[13145.086088] mmc1: Skipping voltage switch <4>[13145.645941] mmc1: Skipping voltage switch <3>[13146.153969] INFO: task halt:30352 blocked for more than 122 seconds. ... Fixes: f2119df6b764 ("mmc: sd: add support for signal voltage switch procedure") Cc: <stable(a)vger.kernel.org> Signed-off-by: Brian Norris <briannorris(a)chromium.org> Reviewed-by: Guenter Roeck <linux(a)roeck-us.net> Link: https://lore.kernel.org/r/20220914014010.2076169-1-briannorris@chromium.org Signed-off-by: Ulf Hansson <ulf.hansson(a)linaro.org> diff --git a/drivers/mmc/core/sd.c b/drivers/mmc/core/sd.c index 06aa62ce0ed1..3662bf5320ce 100644 --- a/drivers/mmc/core/sd.c +++ b/drivers/mmc/core/sd.c @@ -870,7 +870,8 @@ int mmc_sd_get_cid(struct mmc_host *host, u32 ocr, u32 *cid, u32 *rocr) * the CCS bit is set as well. We deliberately deviate from the spec in * regards to this, which allows UHS-I to be supported for SDSC cards. */ - if (!mmc_host_is_spi(host) && rocr && (*rocr & SD_ROCR_S18A)) { + if (!mmc_host_is_spi(host) && (ocr & SD_OCR_S18R) && + rocr && (*rocr & SD_ROCR_S18A)) { err = mmc_set_uhs_voltage(host, pocr); if (err == -EAGAIN) { retries--;

2 years, 2 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

Linux-stable-mirror October 2022