Re: [Linaro-mm-sig] CMA allocation issue : some pages can't be migrated

Hi,

On 8/30/2012 8:45 AM, Aubertin, Guillaume wrote:

looping-in the linaro-mm-sig ML.

On Thu, Aug 30, 2012 at 4:47 PM, Aubertin, Guillaume <g-aubertin@ti.com mailto:g-aubertin@ti.com> wrote:

hi guys,

I've been working for a few days on getting a proper rmmod with the
remoteproc/rpmsg modules, and I stumbled upon an interesting issue.

when doing sucessive memory allocation and release in the CMA
reservation (by loading/unloading the firmware several times),  the
following message shows up  :

[  119.908477] cma: dma_alloc_from_contiguous(cma ed10ad00, count
256, align 8)
[  119.908843] cma: dma_alloc_from_contiguous(): memory range at
c0dfb000 is busy, retrying
[  119.909698] cma: dma_alloc_from_contiguous(): returned c0dfd000

dma_alloc_from_contiguous() tries to allocate the following range,
0xc0dfd000, succesfully this time.

In some cases, the allocation fails after trying several ranges :

[  119.912231] cma: dma_alloc_from_contiguous(cma ed10ad00, count
768, align 8)
[  119.912719] cma: dma_alloc_from_contiguous(): memory range at
c0dff000 is busy, retrying
[  119.913055] cma: dma_alloc_from_contiguous(): memory range at
c0e01000 is busy, retrying
[  119.913055] rproc remoteproc0: dma_alloc_coherent failed: 3145728

Here is my understanding so far :

First, even if we made a CMA reservation, the kernel can still
allocate pages in this area, but these pages must be movable (user
process page by example).

When dma_alloc_from_contiguous() is called to allocate X pages, it
looks for the next X contiguous free pages in it's CMA bitmap (with
respect to the memory alignment). Then, alloc_contig_range() is
called to allocate the given range of pages.  Alloc_contig_range()
analyses the pages we want to allocate, and if a page is already
used, it is migrated to a new page outside the page array we want to
reserve. this is done using isolate_migratepages_range() to list the
pages to migrate, and migrate_pages() to try to migrate the pages,
and that's where it fails. Below is a list of next function calls :

  fallback_migrate_page() --> migrate_page()
  --> try_to_release_page()  --> try_to_free_buffer() -->
drop_buffers() --> buffer_busy()

I understand here that the page contains used buffers that can't be
dropped, and so the page can't be migrated. Well, I must admit that
once here, I'm feeling a little lost in this ocean of memory
management code ;). After a few researches, I found the following
thread on the linux-arm-kernel ML talking about the same issue :

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-June/102844.html with
the following patch :

/ mm/page_alloc.c |    3 ++-/
/ 1 files changed, 2 insertions(+), 1 deletions(-)/
/
/
/diff --git a/mm/page_alloc.c b/mm/page_alloc.c/
/index 0e1c6f5..c9a6483 100644/
/--- a/mm/page_alloc.c/
/+++ b/mm/page_alloc.c/
/@@ -1310,7 +1310,8 @@ void free_hot_cold_page(struct page *page,
int cold)/
/* excessively into the page allocator/
/*//
/if (migratetype >= MIGRATE_PCPTYPES) {/
/-if (unlikely(migratetype == MIGRATE_ISOLATE)) {/
/+if (unlikely(migratetype == MIGRATE_ISOLATE)/
/+  || is_migrate_cma(migratetype)) {/
/free_one_page(zone, page, 0, migratetype);/
/goto out;/
/}/

I tried the patch, and it seems to work (I didn't have any "memory
range busy" in 5000+ tests), but I'm affraid that this could have
some nasty side effects.

Any idea ?

Thanks in advance,
Guillaume

Hi,

Speaking as the author of that patch, I agree that it does have some nasty side effects and is not the right approach. I finally got down to debugging where the extra references came from and asked about it last night (http://lists.linaro.org/pipermail/linaro-mm-sig/2012-August/002510.html)

Basically, as long as the page cache buffers exist on the LRU list they can't be migrated away. I think the fix should be to drop the buffers from the LRU list when migrating away, but as mentioned there I don't really know the filesystem layer well enough to know if that is the right approach.

Unfortunately all the suggested fixes I've seen have nasty side effects so I don't think there has been any consensus on a good solution.

Thanks, Laura

---

Linaro-mm-sig mailing list Linaro-mm-sig@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-mm-sig