Linaro-mm-sig

linaro-mm-sig@lists.linaro.org

15 participants
3030 discussions

[PATCH/RFC 0/3] ARM: DMA-mapping: new extensions for buffer sharing

by Marek Szyprowski

Hello, This patch series introduces a new features to DMA mapping subsystem to let drivers share the allocated buffers (preferably using recently introduced dma_buf framework) easy and efficient. The first extension is DMA_ATTR_NO_KERNEL_MAPPING attribute. It is intended for use with dma_{alloc, mmap, free}_attrs functions. It can be used to notify dma-mapping core that the driver will not use kernel mapping for the allocated buffer at all, so the core can skip creating it. This saves precious kernel virtual address space. Such buffer can be accessed from userspace, after calling dma_mmap_attrs() for it (a typical use case for multimedia buffers). The value returned by dma_alloc_attrs() with this attribute should be considered as a DMA cookie, which needs to be passed to dma_mmap_attrs() and dma_free_attrs() funtions. The second extension is required to let drivers to share the buffers allocated by DMA-mapping subsystem. Right now the driver gets a dma address of the allocated buffer and the kernel virtual mapping for it. If it wants to share it with other device (= map into its dma address space) it usually hacks around kernel virtual addresses to get pointers to pages or assumes that both devices share the DMA address space. Both solutions are just hacks for the special cases, which should be avoided in the final version of buffer sharing. To solve this issue in a generic way, a new call to DMA mapping has been introduced - dma_get_sgtable(). It allocates a scatter-list which describes the allocated buffer and lets the driver(s) to use it with other device(s) by calling dma_map_sg() on it. The proposed patches have been generated on top of the ARM DMA-mapping redesign patch series on Linux v3.4-rc7. They are also available on the following GIT branch: git://git.linaro.org/people/mszyprowski/linux-dma-mapping.git 3.4-rc7-arm-dma-v10-ext with all require patches on top of vanilla v3.4-rc7 kernel. Best regards Marek Szyprowski Samsung Poland R&D Center Patch summary: Marek Szyprowski (3): common: DMA-mapping: add DMA_ATTR_NO_KERNEL_MAPPING attribute ARM: dma-mapping: add support for DMA_ATTR_NO_KERNEL_MAPPING attribute ARM: dma-mapping: add support for dma_get_sgtable() Documentation/DMA-attributes.txt | 18 +++++++++++++ arch/arm/include/asm/dma-mapping.h | 12 +++++++++ arch/arm/mm/dma-mapping.c | 51 ++++++++++++++++++++++++++++++++---- include/linux/dma-attrs.h | 1 + include/linux/dma-mapping.h | 3 +++ 5 files changed, 80 insertions(+), 5 deletions(-) -- 1.7.10.1

13 years, 4 months

[PATCHv2 updated] ARM: dma-mapping: remove custom consistent dma region

by Marek Szyprowski

This patch changes dma-mapping subsystem to use generic vmalloc areas for all consistent dma allocations. This increases the total size limit of the consistent allocations and removes platform hacks and a lot of duplicated code. Atomic allocations are served from special pool preallocated on boot, becasue vmalloc areas cannot be reliably created in atomic context. Signed-off-by: Marek Szyprowski <m.szyprowski(a)samsung.com> --- Hello, This is an updated version of the patch posted in the following thread: http://www.spinics.net/lists/kernel/msg1342885.html This one has been rebased onto the ARM DMA-mapping redesign patches and includes a part for IOMMU-aware ARM DMA-mapping implementation. The ARM DMA-mapping redesign patches are available in the following thread: http://www.spinics.net/lists/arm-kernel/msg175729.html Best regards Marek Szyprowski Samsung Poland R&D Center --- Documentation/kernel-parameters.txt | 4 + arch/arm/include/asm/dma-mapping.h | 2 +- arch/arm/mm/dma-mapping.c | 497 ++++++++++++++++------------------- 3 files changed, 228 insertions(+), 275 deletions(-) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index c1601e5..ba58f50 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -515,6 +515,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted. a hypervisor. Default: yes + coherent_pool=nn[KMG] [ARM,KNL] + Sets the size of memory pool for coherent, atomic dma + allocations. + code_bytes [X86] How many bytes of object code to print in an oops report. Range: 0 - 8192 diff --git a/arch/arm/include/asm/dma-mapping.h b/arch/arm/include/asm/dma-mapping.h index bbef15d..80777d87 100644 --- a/arch/arm/include/asm/dma-mapping.h +++ b/arch/arm/include/asm/dma-mapping.h @@ -226,7 +226,7 @@ static inline int dma_mmap_writecombine(struct device *dev, struct vm_area_struc * DMA region above it's default value of 2MB. It must be called before the * memory allocator is initialised, i.e. before any core_initcall. */ -extern void __init init_consistent_dma_size(unsigned long size); +static inline void init_consistent_dma_size(unsigned long size) { } /* * For SA-1111, IXP425, and ADI systems the dma-mapping functions are "magic" diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 3ac4760..2e98403 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -20,6 +20,7 @@ #include <linux/highmem.h> #include <linux/slab.h> #include <linux/iommu.h> +#include <linux/io.h> #include <linux/vmalloc.h> #include <asm/memory.h> @@ -228,204 +229,170 @@ static void __dma_free_buffer(struct page *page, size_t size) } #ifdef CONFIG_MMU - -#define CONSISTENT_OFFSET(x) (((unsigned long)(x) - consistent_base) >> PAGE_SHIFT) -#define CONSISTENT_PTE_INDEX(x) (((unsigned long)(x) - consistent_base) >> PMD_SHIFT) - -/* - * These are the page tables (2MB each) covering uncached, DMA consistent allocations - */ -static pte_t **consistent_pte; - -#define DEFAULT_CONSISTENT_DMA_SIZE SZ_2M - -unsigned long consistent_base = CONSISTENT_END - DEFAULT_CONSISTENT_DMA_SIZE; - -void __init init_consistent_dma_size(unsigned long size) -{ - unsigned long base = CONSISTENT_END - ALIGN(size, SZ_2M); - - BUG_ON(consistent_pte); /* Check we're called before DMA region init */ - BUG_ON(base < VMALLOC_END); - - /* Grow region to accommodate specified size */ - if (base < consistent_base) - consistent_base = base; -} - -#include "vmregion.h" - -static struct arm_vmregion_head consistent_head = { - .vm_lock = __SPIN_LOCK_UNLOCKED(&consistent_head.vm_lock), - .vm_list = LIST_HEAD_INIT(consistent_head.vm_list), - .vm_end = CONSISTENT_END, -}; - #ifdef CONFIG_HUGETLB_PAGE #error ARM Coherent DMA allocator does not (yet) support huge TLB #endif -/* - * Initialise the consistent memory allocation. - */ -static int __init consistent_init(void) -{ - int ret = 0; - pgd_t *pgd; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - int i = 0; - unsigned long base = consistent_base; - unsigned long num_ptes = (CONSISTENT_END - base) >> PMD_SHIFT; - - consistent_pte = kmalloc(num_ptes * sizeof(pte_t), GFP_KERNEL); - if (!consistent_pte) { - pr_err("%s: no memory\n", __func__); - return -ENOMEM; - } - - pr_debug("DMA memory: 0x%08lx - 0x%08lx:\n", base, CONSISTENT_END); - consistent_head.vm_start = base; - - do { - pgd = pgd_offset(&init_mm, base); - - pud = pud_alloc(&init_mm, pgd, base); - if (!pud) { - pr_err("%s: no pud tables\n", __func__); - ret = -ENOMEM; - break; - } - - pmd = pmd_alloc(&init_mm, pud, base); - if (!pmd) { - pr_err("%s: no pmd tables\n", __func__); - ret = -ENOMEM; - break; - } - WARN_ON(!pmd_none(*pmd)); - - pte = pte_alloc_kernel(pmd, base); - if (!pte) { - pr_err("%s: no pte tables\n", __func__); - ret = -ENOMEM; - break; - } - - consistent_pte[i++] = pte; - base += PMD_SIZE; - } while (base < CONSISTENT_END); - - return ret; -} - -core_initcall(consistent_init); - static void * __dma_alloc_remap(struct page *page, size_t size, gfp_t gfp, pgprot_t prot, const void *caller) { - struct arm_vmregion *c; - size_t align; - int bit; + struct vm_struct *area; + unsigned long addr; - if (!consistent_pte) { - pr_err("%s: not initialised\n", __func__); + area = get_vm_area_caller(size, VM_DMA | VM_USERMAP, caller); + if (!area) + return NULL; + addr = (unsigned long)area->addr; + area->phys_addr = __pfn_to_phys(page_to_pfn(page)); + + if (ioremap_page_range(addr, addr + size, area->phys_addr, prot)) { + vunmap((void *)addr); + return NULL; + } + return (void *)addr; +} + +static void __dma_free_remap(void *cpu_addr, size_t size) +{ + struct vm_struct *area; + + read_lock(&vmlist_lock); + area = find_vm_area(cpu_addr); + if (!area) { + pr_err("%s: trying to free invalid coherent area: %p\n", + __func__, cpu_addr); + dump_stack(); + read_unlock(&vmlist_lock); + return; + } + unmap_kernel_range((unsigned long)cpu_addr, size); + read_unlock(&vmlist_lock); + vunmap(cpu_addr); +} + +struct dma_pool { + size_t size; + spinlock_t lock; + unsigned long *bitmap; + unsigned long count; + void *vaddr; + struct page *page; +}; + +static struct dma_pool atomic_pool = { + .size = SZ_256K, +}; + +static int __init early_coherent_pool(char *p) +{ + atomic_pool.size = memparse(p, &p); + return 0; +} +early_param("coherent_pool", early_coherent_pool); + +/* + * Initialise the coherent pool for atomic allocations. + */ +static int __init atomic_pool_init(void) +{ + struct dma_pool *pool = &atomic_pool; + pgprot_t prot = pgprot_dmacoherent(pgprot_kernel); + unsigned long count = pool->size >> PAGE_SHIFT; + gfp_t gfp = GFP_KERNEL | GFP_DMA; + unsigned long *bitmap; + struct page *page; + void *ptr; + int bitmap_size = BITS_TO_LONGS(count) * sizeof(long); + + bitmap = kzalloc(bitmap_size, GFP_KERNEL); + if (!bitmap) + goto no_bitmap; + + page = __dma_alloc_buffer(NULL, pool->size, gfp); + if (!page) + goto no_page; + + ptr = __dma_alloc_remap(page, pool->size, gfp, prot, NULL); + if (ptr) { + spin_lock_init(&pool->lock); + pool->vaddr = ptr; + pool->page = page; + pool->bitmap = bitmap; + pool->count = count; + pr_info("DMA: preallocated %u KiB pool for atomic coherent allocations\n", + (unsigned)pool->size / 1024); + return 0; + } + + __dma_free_buffer(page, pool->size); +no_page: + kfree(bitmap); +no_bitmap: + pr_err("DMA: failed to allocate %u KiB pool for atomic coherent allocation\n", + (unsigned)pool->size / 1024); + return -ENOMEM; +} +core_initcall(atomic_pool_init); + +static void *__alloc_from_pool(size_t size, struct page **ret_page) +{ + struct dma_pool *pool = &atomic_pool; + unsigned int count = size >> PAGE_SHIFT; + unsigned int pageno; + unsigned long flags; + void *ptr = NULL; + size_t align; + + if (!pool->vaddr) { + pr_err("%s: coherent pool not initialised!\n", __func__); dump_stack(); return NULL; } /* - * Align the virtual region allocation - maximum alignment is - * a section size, minimum is a page size. This helps reduce - * fragmentation of the DMA space, and also prevents allocations - * smaller than a section from crossing a section boundary. + * Align the region allocation - allocations from pool are rather + * small, so align them to their order in pages, minimum is a page + * size. This helps reduce fragmentation of the DMA space. */ - bit = fls(size - 1); - if (bit > SECTION_SHIFT) - bit = SECTION_SHIFT; - align = 1 << bit; + align = PAGE_SIZE << get_order(size); - /* - * Allocate a virtual address in the consistent mapping region. - */ - c = arm_vmregion_alloc(&consistent_head, align, size, - gfp & ~(__GFP_DMA | __GFP_HIGHMEM), caller); - if (c) { - pte_t *pte; - int idx = CONSISTENT_PTE_INDEX(c->vm_start); - u32 off = CONSISTENT_OFFSET(c->vm_start) & (PTRS_PER_PTE-1); - - pte = consistent_pte[idx] + off; - c->priv = page; - - do { - BUG_ON(!pte_none(*pte)); - - set_pte_ext(pte, mk_pte(page, prot), 0); - page++; - pte++; - off++; - if (off >= PTRS_PER_PTE) { - off = 0; - pte = consistent_pte[++idx]; - } - } while (size -= PAGE_SIZE); - - dsb(); - - return (void *)c->vm_start; + spin_lock_irqsave(&pool->lock, flags); + pageno = bitmap_find_next_zero_area(pool->bitmap, pool->count, + 0, count, (1 << align) - 1); + if (pageno < pool->count) { + bitmap_set(pool->bitmap, pageno, count); + ptr = pool->vaddr + PAGE_SIZE * pageno; + *ret_page = pool->page + pageno; } - return NULL; + spin_unlock_irqrestore(&pool->lock, flags); + + return ptr; } -static void __dma_free_remap(void *cpu_addr, size_t size) +static int __free_from_pool(void *start, size_t size) { - struct arm_vmregion *c; - unsigned long addr; - pte_t *ptep; - int idx; - u32 off; + struct dma_pool *pool = &atomic_pool; + unsigned long pageno, count; + unsigned long flags; - c = arm_vmregion_find_remove(&consistent_head, (unsigned long)cpu_addr); - if (!c) { - pr_err("%s: trying to free invalid coherent area: %p\n", - __func__, cpu_addr); + if (start < pool->vaddr || start > pool->vaddr + pool->size) + return 0; + + if (start + size > pool->vaddr + pool->size) { + pr_err("%s: freeing wrong coherent size from pool\n", __func__); dump_stack(); - return; + return 0; } - if ((c->vm_end - c->vm_start) != size) { - pr_err("%s: freeing wrong coherent size (%ld != %d)\n", - __func__, c->vm_end - c->vm_start, size); - dump_stack(); - size = c->vm_end - c->vm_start; - } + pageno = (start - pool->vaddr) >> PAGE_SHIFT; + count = size >> PAGE_SHIFT; - idx = CONSISTENT_PTE_INDEX(c->vm_start); - off = CONSISTENT_OFFSET(c->vm_start) & (PTRS_PER_PTE-1); - ptep = consistent_pte[idx] + off; - addr = c->vm_start; - do { - pte_t pte = ptep_get_and_clear(&init_mm, addr, ptep); + spin_lock_irqsave(&pool->lock, flags); + bitmap_clear(pool->bitmap, pageno, count); + spin_unlock_irqrestore(&pool->lock, flags); - ptep++; - addr += PAGE_SIZE; - off++; - if (off >= PTRS_PER_PTE) { - off = 0; - ptep = consistent_pte[++idx]; - } - - if (pte_none(pte) || !pte_present(pte)) - pr_crit("%s: bad page in kernel page table\n", - __func__); - } while (size -= PAGE_SIZE); - - flush_tlb_kernel_range(c->vm_start, c->vm_end); - - arm_vmregion_free(&consistent_head, c); + return 1; } static inline pgprot_t __get_dma_pgprot(struct dma_attrs *attrs, pgprot_t prot) @@ -441,6 +408,8 @@ static inline pgprot_t __get_dma_pgprot(struct dma_attrs *attrs, pgprot_t prot) #define __dma_alloc_remap(page, size, gfp, prot, c) page_address(page) #define __dma_free_remap(addr, size) do { } while (0) #define __get_dma_pgprot(attrs, prot) __pgprot(0) +#define __alloc_from_pool(size, ret_page) NULL +#define __free_from_pool(addr, size) 0 #endif /* CONFIG_MMU */ @@ -463,6 +432,16 @@ __dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, *handle = DMA_ERROR_CODE; size = PAGE_ALIGN(size); + /* + * Atomic allocations need special handling + */ + if (gfp & GFP_ATOMIC && !arch_is_coherent()) { + addr = __alloc_from_pool(size, &page); + if (addr) + *handle = pfn_to_dma(dev, page_to_pfn(page)); + return addr; + } + page = __dma_alloc_buffer(dev, size, gfp); if (!page) return NULL; @@ -506,30 +485,21 @@ int arm_dma_mmap(struct device *dev, struct vm_area_struct *vma, { int ret = -ENXIO; #ifdef CONFIG_MMU - unsigned long user_size, kern_size; - struct arm_vmregion *c; + unsigned long user_count = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; + unsigned long pfn = dma_to_pfn(dev, dma_addr); + unsigned long off = vma->vm_pgoff; vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot); if (dma_mmap_from_coherent(dev, vma, cpu_addr, size, &ret)) return ret; - user_size = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; - - c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr); - if (c) { - unsigned long off = vma->vm_pgoff; - struct page *pages = c->priv; - - kern_size = (c->vm_end - c->vm_start) >> PAGE_SHIFT; - - if (off < kern_size && - user_size <= (kern_size - off)) { - ret = remap_pfn_range(vma, vma->vm_start, - page_to_pfn(pages) + off, - user_size << PAGE_SHIFT, - vma->vm_page_prot); - } + if (off < count && user_count <= (count - off)) { + ret = remap_pfn_range(vma, vma->vm_start, + pfn + off, + user_count << PAGE_SHIFT, + vma->vm_page_prot); } #endif /* CONFIG_MMU */ @@ -543,13 +513,16 @@ int arm_dma_mmap(struct device *dev, struct vm_area_struct *vma, void arm_dma_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t handle, struct dma_attrs *attrs) { - WARN_ON(irqs_disabled()); - if (dma_release_from_coherent(dev, get_order(size), cpu_addr)) return; size = PAGE_ALIGN(size); + if (__free_from_pool(cpu_addr, size)) + return; + + WARN_ON(irqs_disabled()); + if (!arch_is_coherent()) __dma_free_remap(cpu_addr, size); @@ -769,9 +742,6 @@ static int arm_dma_set_mask(struct device *dev, u64 dma_mask) static int __init dma_debug_do_init(void) { -#ifdef CONFIG_MMU - arm_vmregion_create_proc("dma-mappings", &consistent_head); -#endif dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); return 0; } @@ -888,61 +858,30 @@ static int __iommu_free_buffer(struct device *dev, struct page **pages, size_t s * Create a CPU mapping for a specified pages */ static void * -__iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot) +__iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot, + const void *caller) { - struct arm_vmregion *c; - size_t align; - size_t count = size >> PAGE_SHIFT; - int bit; + unsigned int i, count = size >> PAGE_SHIFT; + struct vm_struct *area; + unsigned long p; - if (!consistent_pte[0]) { - pr_err("%s: not initialised\n", __func__); - dump_stack(); + area = get_vm_area_caller(size, VM_DMA | VM_USERMAP, caller); + if (!area) return NULL; + + area->pages = pages; + p = (unsigned long)area->addr; + + for (i = 0; i < count; i++) { + phys_addr_t phys = __pfn_to_phys(page_to_pfn(pages[i])); + if (ioremap_page_range(p, p + PAGE_SIZE, phys, prot)) + goto err; + p += PAGE_SIZE; } - - /* - * Align the virtual region allocation - maximum alignment is - * a section size, minimum is a page size. This helps reduce - * fragmentation of the DMA space, and also prevents allocations - * smaller than a section from crossing a section boundary. - */ - bit = fls(size - 1); - if (bit > SECTION_SHIFT) - bit = SECTION_SHIFT; - align = 1 << bit; - - /* - * Allocate a virtual address in the consistent mapping region. - */ - c = arm_vmregion_alloc(&consistent_head, align, size, - gfp & ~(__GFP_DMA | __GFP_HIGHMEM), NULL); - if (c) { - pte_t *pte; - int idx = CONSISTENT_PTE_INDEX(c->vm_start); - int i = 0; - u32 off = CONSISTENT_OFFSET(c->vm_start) & (PTRS_PER_PTE-1); - - pte = consistent_pte[idx] + off; - c->priv = pages; - - do { - BUG_ON(!pte_none(*pte)); - - set_pte_ext(pte, mk_pte(pages[i], prot), 0); - pte++; - off++; - i++; - if (off >= PTRS_PER_PTE) { - off = 0; - pte = consistent_pte[++idx]; - } - } while (i < count); - - dsb(); - - return (void *)c->vm_start; - } + return area->addr; +err: + unmap_kernel_range((unsigned long)area->addr, size); + vunmap(area->addr); return NULL; } @@ -1001,6 +940,17 @@ static int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t si return 0; } +static struct page **__iommu_get_pages(void *cpu_addr) +{ + struct vm_struct *area; + read_lock(&vmlist_lock); + area = find_vm_area(cpu_addr); + read_unlock(&vmlist_lock); + if (area) + return area->pages; + return NULL; +} + static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, struct dma_attrs *attrs) { @@ -1019,7 +969,8 @@ static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, if (*handle == DMA_ERROR_CODE) goto err_buffer; - addr = __iommu_alloc_remap(pages, size, gfp, prot); + addr = __iommu_alloc_remap(pages, size, gfp, prot, + __builtin_return_address(0)); if (!addr) goto err_mapping; @@ -1036,31 +987,25 @@ static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, struct dma_attrs *attrs) { - struct arm_vmregion *c; + unsigned long uaddr = vma->vm_start; + unsigned long usize = vma->vm_end - vma->vm_start; + struct page **pages = __iommu_get_pages(cpu_addr); vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot); - c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr); - if (c) { - struct page **pages = c->priv; + if (!pages) + return -ENXIO; - unsigned long uaddr = vma->vm_start; - unsigned long usize = vma->vm_end - vma->vm_start; - int i = 0; + do { + int ret = vm_insert_page(vma, uaddr, *pages++); + if (ret) { + pr_err("Remapping memory failed: %d\n", ret); + return ret; + } + uaddr += PAGE_SIZE; + usize -= PAGE_SIZE; + } while (usize > 0); - do { - int ret; - - ret = vm_insert_page(vma, uaddr, pages[i++]); - if (ret) { - pr_err("Remapping memory, error: %d\n", ret); - return ret; - } - - uaddr += PAGE_SIZE; - usize -= PAGE_SIZE; - } while (usize > 0); - } return 0; } @@ -1071,16 +1016,20 @@ static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, void arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, dma_addr_t handle, struct dma_attrs *attrs) { - struct arm_vmregion *c; + struct page **pages = __iommu_get_pages(cpu_addr); size = PAGE_ALIGN(size); - c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr); - if (c) { - struct page **pages = c->priv; - __dma_free_remap(cpu_addr, size); - __iommu_remove_mapping(dev, handle, size); - __iommu_free_buffer(dev, pages, size); + if (!pages) { + pr_err("%s: trying to free invalid coherent area: %p\n", + __func__, cpu_addr); + dump_stack(); + return; } + + unmap_kernel_range((unsigned long)cpu_addr, size); + vunmap(cpu_addr); + __iommu_remove_mapping(dev, handle, size); + __iommu_free_buffer(dev, pages, size); } /* -- 1.7.10.1

13 years, 4 months

[PATCHv10 00/11] ARM: DMA-mapping framework redesign

by Marek Szyprowski

Hello, This is another update on dma-mapping redesign patches for ARM. I integrated a few minor fixes that were needed to solve the issues reported after putting these patches for testing in linux-next branch. The patches are also available on my git repository at: git://git.linaro.org/people/mszyprowski/linux-dma-mapping.git 3.4-rc7-arm-dma-v10 History of the development: v1: (initial version of the DMA-mapping redesign patches): http://www.spinics.net/lists/linux-mm/msg21241.html v2: http://lists.linaro.org/pipermail/linaro-mm-sig/2011-September/000571.html http://lists.linaro.org/pipermail/linaro-mm-sig/2011-September/000577.html v3: http://www.spinics.net/lists/linux-mm/msg25490.html v4 and v5: http://www.spinics.net/lists/arm-kernel/msg151147.html http://www.spinics.net/lists/arm-kernel/msg154889.html v6: http://www.spinics.net/lists/linux-mm/msg29903.html v7: http://www.spinics.net/lists/arm-kernel/msg162149.html v8: http://www.spinics.net/lists/arm-kernel/msg168478.html v9: http://www.spinics.net/lists/linux-arch/msg17443.html Best regards Marek Szyprowski Samsung Poland R&D Center Patch summary: Marek Szyprowski (11): common: add dma_mmap_from_coherent() function ARM: dma-mapping: use dma_mmap_from_coherent() ARM: dma-mapping: use pr_* instread of printk ARM: dma-mapping: introduce DMA_ERROR_CODE constant ARM: dma-mapping: remove offset parameter to prepare for generic dma_ops ARM: dma-mapping: use asm-generic/dma-mapping-common.h ARM: dma-mapping: implement dma sg methods on top of any generic dma ops ARM: dma-mapping: move all dma bounce code to separate dma ops structure ARM: dma-mapping: remove redundant code and do the cleanup ARM: dma-mapping: use alloc, mmap, free from dma_ops ARM: dma-mapping: add support for IOMMU mapper arch/arm/Kconfig | 9 + arch/arm/common/dmabounce.c | 84 ++- arch/arm/include/asm/device.h | 4 + arch/arm/include/asm/dma-iommu.h | 34 ++ arch/arm/include/asm/dma-mapping.h | 407 ++++----------- arch/arm/mm/dma-mapping.c | 1015 ++++++++++++++++++++++++++++++------ arch/arm/mm/vmregion.h | 2 +- drivers/base/dma-coherent.c | 42 ++ include/asm-generic/dma-coherent.h | 4 +- 9 files changed, 1134 insertions(+), 467 deletions(-) create mode 100644 arch/arm/include/asm/dma-iommu.h -- 1.7.10.1

13 years, 4 months

[PATCH] ARM: LPAE: fix access flag setup in mem_type_table

by Vitaly Andrianov

A zero value for prot_sect in the memory types table implies that section mappings should never be created for the memory type in question. This is checked for in alloc_init_section(). With LPAE, we set a bit to mask access flag faults for kernel mappings. This breaks the aforementioned (!prot_sect) check in alloc_init_section(). This patch fixes this bug by first checking for a non-zero prot_sect before setting the PMD_SECT_AF flag. Signed-off-by: Vitaly Andrianov <vitalya(a)ti.com> Acked-by: Catalin Marinas <catalin.marinas(a)arm.com> --- arch/arm/mm/mmu.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index b9fbec2..1e16b20 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -494,7 +494,8 @@ static void __init build_mem_type_table(void) */ for (i = 0; i < ARRAY_SIZE(mem_types); i++) { mem_types[i].prot_pte |= PTE_EXT_AF; - mem_types[i].prot_sect |= PMD_SECT_AF; + if (mem_types[i].prot_sect) + mem_types[i].prot_sect |= PMD_SECT_AF; } kern_pgprot |= PTE_EXT_AF; vecs_pgprot |= PTE_EXT_AF; -- 1.7.5.4

13 years, 4 months

[PATCH] ARM: LPAE: fix access flag setup in mem_type_table

by Vitaly Andrianov

13 years, 4 months

[PATCH] ARM: dma-mapping: use PMD size for section unmap

by Vitaly Andrianov

The dma_contiguous_remap() function clears existing section maps using the wrong size (PGDIR_SIZE instead of PMD_SIZE). This is a bug which does not affect non-LPAE systems, where PGDIR_SIZE and PMD_SIZE are the same. On LPAE systems, however, this bug causes the kernel to hang at this point. This fix has been tested on both LPAE and non-LPAE kernel builds. Signed-off-by: Vitaly Andrianov <vitalya(a)ti.com> --- arch/arm/mm/dma-mapping.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 446dc1b..d220d4f 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -414,7 +414,7 @@ void __init dma_contiguous_remap(void) * Clear previous low-memory mapping */ for (addr = __phys_to_virt(start); addr < __phys_to_virt(end); - addr += PGDIR_SIZE) + addr += PMD_SIZE) pmd_clear(pmd_off_k(addr)); iotable_init(&map, 1); -- 1.7.5.4

13 years, 4 months

[PATCH 3/3] [RFC]:DMA-MAPPING:Add check inside IOMMU ops for kernel or user space allcoation

by Abhinav

With this we can do a run time check on the allocation type for either kernel or user using the dma attribute passed to dma-mapping iommu ops. Signed-off-by: Abhinav <abhinav.k(a)samsung.com> --- arch/arm/mm/dma-mapping.c | 88 +++++++++++++++++++++++++++++++++++---------- 1 files changed, 69 insertions(+), 19 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 2c5a285..4cd46b4 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -428,6 +428,7 @@ static void __dma_free_remap(void *cpu_addr, size_t size) arm_vmregion_free(&consistent_head, c); } + #else /* !CONFIG_MMU */ #define __dma_alloc_remap(page, size, gfp, prot, c) page_address(page) @@ -894,6 +895,35 @@ __iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot) size_t align; size_t count = size >> PAGE_SHIFT; int bit; + unsigned long mem_type = (unsigned long)gfp; + + + if(mem_type){ + + struct page_infodma *pages_in; + + pages_in = kzalloc( sizeof(struct page_infodma*), GFP_KERNEL); + if(!pages_in) + return NULL; + + pages_in->nr_pages = count; + + return (void*)pages_in; + + } + + /* + * Align the virtual region allocation - maximum alignment is + * a section size, minimum is a page size. This helps reduce + * fragmentation of the DMA space, and also prevents allocations + * smaller than a section from crossing a section boundary. + */ + + bit = fls(size - 1); + if (bit > SECTION_SHIFT) + bit = SECTION_SHIFT; + align = 1 << bit; + if (!consistent_pte[0]) { pr_err("%s: not initialised\n", __func__); @@ -901,16 +931,6 @@ __iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot) return NULL; } - /* - * Align the virtual region allocation - maximum alignment is - * a section size, minimum is a page size. This helps reduce - * fragmentation of the DMA space, and also prevents allocations - * smaller than a section from crossing a section boundary. - */ - bit = fls(size - 1); - if (bit > SECTION_SHIFT) - bit = SECTION_SHIFT; - align = 1 << bit; /* * Allocate a virtual address in the consistent mapping region. @@ -946,6 +966,7 @@ __iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot) return NULL; } + /* * Create a mapping in device IO address space for specified pages */ @@ -973,13 +994,16 @@ __iommu_create_mapping(struct device *dev, struct page **pages, size_t size) len = (j - i) << PAGE_SHIFT; ret = iommu_map(mapping->domain, iova, phys, len, 0); + if (ret < 0) goto fail; + iova += len; i = j; } return dma_addr; fail: + iommu_unmap(mapping->domain, dma_addr, iova-dma_addr); __free_iova(mapping, dma_addr, size); return DMA_ERROR_CODE; @@ -1007,6 +1031,8 @@ static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, pgprot_t prot = __get_dma_pgprot(attrs, pgprot_kernel); struct page **pages; void *addr = NULL; + struct page_infodma *page_ret; + unsigned long mem_type; *handle = DMA_ERROR_CODE; size = PAGE_ALIGN(size); @@ -1019,11 +1045,19 @@ static void *arm_iommu_alloc_attrs(struct device *dev, size_t size, if (*handle == DMA_ERROR_CODE) goto err_buffer; - addr = __iommu_alloc_remap(pages, size, gfp, prot); + mem_type = dma_get_attr(DMA_ATTR_USER_SPACE, attrs); + + addr = __iommu_alloc_remap(pages, size, mem_type, prot); if (!addr) goto err_mapping; - return addr; + if(mem_type){ + page_ret = (struct page_infodma *)addr; + page_ret->pages = pages; + return page_ret; + } + else + return addr; err_mapping: __iommu_remove_mapping(dev, *handle, size); @@ -1071,18 +1105,34 @@ static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, void arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, dma_addr_t handle, struct dma_attrs *attrs) { - struct arm_vmregion *c; + + unsigned long mem_type = dma_get_attr(DMA_ATTR_USER_SPACE, attrs); + size = PAGE_ALIGN(size); - c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr); - if (c) { - struct page **pages = c->priv; - __dma_free_remap(cpu_addr, size); - __iommu_remove_mapping(dev, handle, size); - __iommu_free_buffer(dev, pages, size); + + if(mem_type){ + + struct page_infodma *pagesin = cpu_addr; + if (pagesin) { + struct page **pages = pagesin->pages; + __iommu_remove_mapping(dev, handle, size); + __iommu_free_buffer(dev, pages, size); + } + } + else{ + struct arm_vmregion *c; + c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr); + if (c) { + struct page **pages = c->priv; + __dma_free_remap(cpu_addr, size); + __iommu_remove_mapping(dev, handle, size); + __iommu_free_buffer(dev, pages, size); + } } } + /* * Map a part of the scatter-gather list into contiguous io address space */ -- 1.7.0.4

13 years, 4 months

[PATCH 2/3] [RFC]:DMA-MAPPING:Define new wrapper functions to pass new dma attribute to IOMMU ops

by Abhinav

With these new functions the drivers can pass the new dma attribute to IOMMU ops of dma mapping framework to differentiate between kernel and user space allocations. Signed-off-by: Abhinav <abhinav.k(a)samsung.com> --- arch/arm/include/asm/dma-mapping.h | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-) diff --git a/arch/arm/include/asm/dma-mapping.h b/arch/arm/include/asm/dma-mapping.h index bbef15d..7fc003a 100644 --- a/arch/arm/include/asm/dma-mapping.h +++ b/arch/arm/include/asm/dma-mapping.h @@ -14,6 +14,12 @@ #define DMA_ERROR_CODE (~0) extern struct dma_map_ops arm_dma_ops; +struct page_infodma { + struct page **pages; + unsigned long nr_pages; + unsigned long shared; +}; + static inline struct dma_map_ops *get_dma_ops(struct device *dev) { if (dev && dev->archdata.dma_ops) @@ -205,6 +211,14 @@ static inline void *dma_alloc_writecombine(struct device *dev, size_t size, return dma_alloc_attrs(dev, size, dma_handle, flag, &attrs); } +static inline void *dma_alloc_writecombine_user(struct device *dev, size_t size, + dma_addr_t *dma_handle, gfp_t flag) +{ + DEFINE_DMA_ATTRS(attrs); + dma_set_attr(DMA_ATTR_USER_SPACE, &attrs); + return dma_alloc_attrs(dev, size, dma_handle, flag, &attrs); +} + static inline void dma_free_writecombine(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_handle) { @@ -213,6 +227,14 @@ static inline void dma_free_writecombine(struct device *dev, size_t size, return dma_free_attrs(dev, size, cpu_addr, dma_handle, &attrs); } +static inline void dma_free_writecombine_user(struct device *dev, size_t size, + void *cpu_addr, dma_addr_t dma_handle) +{ + DEFINE_DMA_ATTRS(attrs); + dma_set_attr(DMA_ATTR_USER_SPACE, &attrs); + return dma_free_attrs(dev, size, cpu_addr, dma_handle, &attrs); +} + static inline int dma_mmap_writecombine(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size) { @@ -221,6 +243,14 @@ static inline int dma_mmap_writecombine(struct device *dev, struct vm_area_struc return dma_mmap_attrs(dev, vma, cpu_addr, dma_addr, size, &attrs); } +static inline int dma_mmap_writecombine_user(struct device *dev, struct vm_area_struct *vma, + void *cpu_addr, dma_addr_t dma_addr, size_t size) +{ + DEFINE_DMA_ATTRS(attrs); + dma_set_attr(DMA_ATTR_USER_SPACE, &attrs); + return dma_mmap_attrs(dev, vma, cpu_addr, dma_addr, size, &attrs); +} + /* * This can be called during boot to increase the size of the consistent * DMA region above it's default value of 2MB. It must be called before the -- 1.7.0.4

13 years, 4 months

[PATCH 1/3] [RFC]:DMA-MAPPING:Add a new DMA attribute to avoid Kernel virtual memory allocation

by Abhinav

This dma attribute can be used to pass to the iommu ops of dma-mapping framework to differentiate between kernel and user space allocations. Signed-off-by: Abhinav <abhinav.k(a)samsung.com> --- include/linux/dma-attrs.h | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/include/linux/dma-attrs.h b/include/linux/dma-attrs.h index 547ab56..861df09 100644 --- a/include/linux/dma-attrs.h +++ b/include/linux/dma-attrs.h @@ -15,6 +15,7 @@ enum dma_attr { DMA_ATTR_WEAK_ORDERING, DMA_ATTR_WRITE_COMBINE, DMA_ATTR_NON_CONSISTENT, + DMA_ATTR_USER_SPACE, DMA_ATTR_MAX, }; -- 1.7.0.4

13 years, 4 months

how to avoid allocating or freeze MOVABLE memory in userspace

by Haojian Zhuang

Hi all, I have one question on memory migration. As we know, malloc() from user app will allocate MIGRATE_MOVABLE pages. But if we want to use this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could we change its behavior before DMA working? Thanks Haojian

13 years, 4 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-mm-sig