On Tue, Apr 19, 2011 at 2:49 PM, Nicolas Pitre
<nicolas.pitre@linaro.org> wrote:
On Tue, 19 Apr 2011, Rebecca Schultz Zavin wrote:
> Hey all,
>
> While we are working out requirements, I was hoping to get some more
> information about another related issue that keeps coming up on mailing
> lists and in discussions.
>
> ARM has stated that if you have the same physical memory mapped with two
> different sets of attribute bits you get undefined behavior. I think it's
> going to be a requirement that some of the memory allocated via the unified
> memory manager is mapped uncached. However, because all of memory is mapped
> cached into the unity map at boot, we already have two mappings with
> different attributes. I want to understand the mechanism of the problem,
> because none of the solutions I can come up with are particularly nice. I'd
> also like to know exactly which architectures are affected, since the fix
> may be costly in performance, memory or both. Can someone at ARM explain to
> me why this causes a problem. I have a theory, but it's mostly a guess.
My own guess is that the cacheable attribute is tied to cache entries
which are physically tagged. Access to one mapping could establish some
attributes that the second mapping would inherit on cache hit.
My guess was that if a line was present in the L2, even accesses via an uncached mapping would get serviced from there. Presumably the prefetcher would have had to access the cached mapping, causing it to get populated into the l2. Then when the uncached mapping is accessed later, we see the old, possibly stale one. The only mechanism by which this makes sense to me is if the cache attributes are only checked when deciding to put a line into the l2 and not when retrieving one.
> I especially want to understand if it's still a problem if we never
> access the memory via the mapping in the unity map. I know
> speculative prefetching is part of the issue, so I assume older
> architectures without that feature don't exhibit this behaviour
Yes, speculative prefetching is what will cause spurious accesses
through the
kernel direct mapping (that's how it is called in Linux) even if you
don't access it explicitly. Older architectures don't have speculative
prefetching, and even older ones have VIVT caches which has no problem
with multiple different mappings.
> If we really need all mappings of physical memory to have the same cache
> attribute bits, I see three workarounds:
>
> 1- set aside memory at boot that never gets mapped by the kernel. The
> unified memory manager can then ensure there's only one mapping at a time.
> Obvious drawbacks here are that you have to statically partition your system
> into memory you want accessible to the unified memory manager and memory you
> don't. This may not be that big a deal, since most current solutions, pmem,
> cmem, et al basically do this. I can say that on Android devices running on
> a high resolution display (720p and above) we're easily talking about
> needing 256M of memory or more to dedicate to this.
This is obviously suboptimal.
> 2- use highmem pages only for the unified memory manager. Highmem pages
> only get mapped on demand.
> This has some performance costs when the kernel allocates other metadata in
> highmem. Most embedded systems still don't have enough memory to need
> highmem, though I'm guessing that'll follow the current trend and shift in
> the next couple of years.
The kernel tries not to allocate its own data in highmem. Instead,
highmem pages are used for user space processes or the buffer cache
which can be populated directly by DMA and be largely untouched by the
kernel. The highmem pages are also fairly easily reclaimable making
them an easy target when large physically contiguous allocations are
required.
It is true that most systems might not have enough memory to require
highmem, but they can make use of it nevertheless, simply by changing
the direct mapped memory threshold.
While highmem is not free in terms of overhead, it is still quite
lightweight compared to other memory partitioning schemes, and above all
it is already supported across the whole kernel and relied upon by many
people already.
This is actually my favorite solution -- it's easy and not to gross. To be honest we've been tweaking things so we get some highmem on some android platforms for a while, because the binder is a huge kernel address space hog. My plan for implementing a memory manager is to start with this and solve the harder problem only if I have to.
> 3- fix up the unity mapping so the attribute bits match those desired by the
> unified memory manger. This could be done by removing pages from the unity
> map. It's complicated by the fact that the unity map makes use of large
> pages, sections and supersections to reduce tlb pressure. I don't think
> this is impossible if we restrict the set of contexts from which it can
> happen, but I'm imagining that we will also need to maintain some kind of
> pool of memory we've moved from cached to uncached since the process is
> likely to be expensive. Quite likely we will have to iterate
> over processes and update all their top level page tables.
The kernel direct mapping share the same mapping entries across all
processes. So if (part of) the kernel direct mapping uses second level
page table entries, then the first level entries will share the same
second level page table across all processes. Hence changing memory
attributes for those pages covered by that second level table won't
require any itteration over all processes. Obviously the drawback here
is more TLB pressure, however if the memory put aside is not used by the
kernel directly then the associated TLBs won't be involved.
I think the whole kernel direct mapping might be in 1st level page tables, assuming it's an integer multiple of sections or supersections and aligned properly. I suppose we could hack things to make sure there were at least 1 supersection's worth of second level page table entries required. I came to the same conclusion about TLB pressure, that it didn't matter much if we don't actually ever use those TLBs anyway. We actually had some patches from folks at nvidia to solve part of this problem by just making the whole kernel direct map page based. We never measured the performance impact, just decided we were uncomfortable with the change.
> These all have drawbacks, so I'd like to really understand the problem
> before pursuing them. Can the linaro folks find someone who can explain the
> problem in more detail?
I'm happy to discuss about the details when needed.
Nicolas
Rebecca