Hi OP-TEE experts,
I'm looking at OP-TEE Secure OS and came across number of questions that I need to clarify:
- It seems OP-TEE is using only L1 page tables
* It has one large L1 page table for kernel which maps flat 4 GB space (referenced by TTBR1); every thread has another L1 translation table(s) referenced by TTBR0 which map 32 MB of virtual addresses
* Does this mean that the smallest unit of memory that it can map is 1 MB?
* If Trusted Application code/data/stack is 16 KB total, does it still consume 1 MB of DRAM?
- TA Memory usage
* Browsing through OP-TEE source (tee_ta_load_user_ta() function) we noticed that heap/stack for all TAs seems to be allocated from the same pool, is this correct?
* If yes, then can different TAs see heap/stack of each other? If this is possible, then what is security model of OP-TEE? Does it assume that all TAs are trusted and don't need isolation from each other? Any high level design description would be very helpful
- Access to HW resources
* If TA needs to access some HW resources, how can this be done? Does OP-TEE has concept of loadable secure Drivers?
- Suspend/Resume
* Does OP-TEE support saving/restoring its state for CPU suspend/resume cycle?
- Scheduling of TA threads
* Does OP-TEE Swd kernel have periodic timer interrupt handler that causes the task switch or does it rely on Nwd making some sort of SMC call for that?
If someone can help clarify this it would be very much appreciated.
Thanks, Sergey Blotsky
Hi Sergey,
On Mon, Aug 10, 2015 at 09:15:40PM +0000, Blotsky, Sergey wrote:
Hi OP-TEE experts,
I'm looking at OP-TEE Secure OS and came across number of questions that I need to clarify:
It seems OP-TEE is using only L1 page tables
It has one large L1 page table for kernel which maps flat 4 GB space (referenced by TTBR1); every thread has another L1 translation table(s) referenced by TTBR0 which map 32 MB of virtual addresses
Yes, that's correct for the v7 translation tables except that we map the TEE Core memory with L2 tables when the pager is enabled. For LPAE it's slightly different.
Does this mean that the smallest unit of memory that it can map is 1 MB?
Yes
If Trusted Application code/data/stack is 16 KB total, does it still consume 1 MB of DRAM?
Yes, but long term this is something we'd like to address with more fine grained mapping.
TA Memory usage
Browsing through OP-TEE source (tee_ta_load_user_ta() function) we noticed that heap/stack for all TAs seems to be allocated from the same pool, is this correct?
They use the same virtual address range, but as each TA instance maps virtual memory to different physical memory they don't share memory.
If yes, then can different TAs see heap/stack of each other? If this is possible, then what is security model of OP-TEE? Does it assume that all TAs are trusted and don't need isolation from each other? Any high level design description would be very helpful
TAs are isolated from each other. There's currently no support for shared secure memory.
Access to HW resources
If TA needs to access some HW resources, how can this be done? Does OP-TEE has concept of loadable secure Drivers?
We don't have a concept of loadable secure drivers. To expose a generic driver to a TA I would recomend defining a Static TA (see examples in core/arch/arm/sta) which can provide an interface to the driver to be used by a TA. It's currently not possible to check if the calls to the Static TA comes from a TA of from normal world, but adding that is very easy as that information is availble in tee_ta_open_session().
Suspend/Resume
Does OP-TEE support saving/restoring its state for CPU suspend/resume cycle?
Currently not, in the ARMv8 case ARM Trusted Firmware all that's needed.
For ARMv7 we haven't had to deal with that yet. OP-TEE current doesn't have any state that need to be saved so the CPU just need to be reinitialized with reset_secondary() in core/arch/arm/kernel/generic_entry_a32.S. The function can probably not be used directly as we may need to deal differently with synchronization between CPUs.
Scheduling of TA threads
Does OP-TEE Swd kernel have periodic timer interrupt handler that causes the task switch or does it rely on Nwd making some sort of SMC call for that?
OP-TEE is scheduled from normal world so we don't use any timer interrupts in secure world for that. If we're executing in secure world when a normal world timer interrupt (or any other for that matter) is triggered we save the secure thread state and exit to normal world. Normal world will get the interrupt as soon as we've switched to normal world as we haven't done anything with the interrupt in secure world. Once the normal world interrupt handler returns we'll switch back to secure world and resume the secure world thread again.
You can find details on interrupt handling at https://docs.google.com/document/d/1-GthGR5alcaJQxzjs76ocI3FldXU4wYalw13VHtR...
Regards, Jens
Hi Jens,
Thanks for your response, it does help to clarify our understanding of OP-TEE.
As for the stack/heap allocation, I was referring to the following code in tee_ta_load_user_ta() function from tee_ta_manager.c:
/* * Allocate heap and stack */ ctx->mm_heap_stack = tee_mm_alloc(&tee_mm_sec_ddr, *heap_size + ctx->stack_size); if (!ctx->mm_heap_stack) { EMSG("Failed to allocate %zu bytes\n", *heap_size + ctx->stack_size); EMSG(" of memory for user heap and stack\n"); return TEE_ERROR_OUT_OF_MEMORY; }
Is this allocating TA stack/heap? It seems to me that stack/heap is allocated from tee_mm_sec_ddr buffer defined in tee_mm_unpg.c:
/* Physical Secure DDR pool */ tee_mm_pool_t tee_mm_sec_ddr __data; /* XXX __data is a workaround */
To me this looks like all stacks are allocated from the same buffer. Perhaps I'm just looking at wrong place, or this is not the actual TA stack being allocated here... It seems I'm just missing something here. Can you please help clarify this?
Thanks, Sergey
-----Original Message----- From: Jens Wiklander [mailto:jens.wiklander@linaro.org] Sent: August-17-15 3:46 AM To: Blotsky, Sergey Cc: tee-dev@lists.linaro.org Subject: Re: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Hi Sergey,
On Mon, Aug 10, 2015 at 09:15:40PM +0000, Blotsky, Sergey wrote:
Hi OP-TEE experts,
I'm looking at OP-TEE Secure OS and came across number of questions that I need to clarify:
It seems OP-TEE is using only L1 page tables
It has one large L1 page table for kernel which maps flat 4 GB space (referenced by TTBR1); every thread has another L1 translation table(s) referenced by TTBR0 which map 32 MB of virtual addresses
Yes, that's correct for the v7 translation tables except that we map the TEE Core memory with L2 tables when the pager is enabled. For LPAE it's slightly different.
Does this mean that the smallest unit of memory that it can map is 1 MB?
Yes
If Trusted Application code/data/stack is 16 KB total, does it still consume 1 MB of DRAM?
Yes, but long term this is something we'd like to address with more fine grained mapping.
TA Memory usage
Browsing through OP-TEE source (tee_ta_load_user_ta() function) we noticed that heap/stack for all TAs seems to be allocated from the same pool, is this correct?
They use the same virtual address range, but as each TA instance maps virtual memory to different physical memory they don't share memory.
If yes, then can different TAs see heap/stack of each other? If this is possible, then what is security model of OP-TEE? Does it assume that all TAs are trusted and don't need isolation from each other? Any high level design description would be very helpful
TAs are isolated from each other. There's currently no support for shared secure memory.
Access to HW resources
If TA needs to access some HW resources, how can this be done? Does OP-TEE has concept of loadable secure Drivers?
We don't have a concept of loadable secure drivers. To expose a generic driver to a TA I would recomend defining a Static TA (see examples in core/arch/arm/sta) which can provide an interface to the driver to be used by a TA. It's currently not possible to check if the calls to the Static TA comes from a TA of from normal world, but adding that is very easy as that information is availble in tee_ta_open_session().
Suspend/Resume
Does OP-TEE support saving/restoring its state for CPU suspend/resume cycle?
Currently not, in the ARMv8 case ARM Trusted Firmware all that's needed.
For ARMv7 we haven't had to deal with that yet. OP-TEE current doesn't have any state that need to be saved so the CPU just need to be reinitialized with reset_secondary() in core/arch/arm/kernel/generic_entry_a32.S. The function can probably not be used directly as we may need to deal differently with synchronization between CPUs.
Scheduling of TA threads
Does OP-TEE Swd kernel have periodic timer interrupt handler that causes the task switch or does it rely on Nwd making some sort of SMC call for that?
OP-TEE is scheduled from normal world so we don't use any timer interrupts in secure world for that. If we're executing in secure world when a normal world timer interrupt (or any other for that matter) is triggered we save the secure thread state and exit to normal world. Normal world will get the interrupt as soon as we've switched to normal world as we haven't done anything with the interrupt in secure world. Once the normal world interrupt handler returns we'll switch back to secure world and resume the secure world thread again.
You can find details on interrupt handling at https://docs.google.com/document/d/1-GthGR5alcaJQxzjs76ocI3FldXU4wYalw13VHtR...
Regards, Jens
Hi,
On Mon, Aug 17, 2015 at 02:26:36PM +0000, Blotsky, Sergey wrote:
Hi Jens,
Thanks for your response, it does help to clarify our understanding of OP-TEE.
As for the stack/heap allocation, I was referring to the following code in tee_ta_load_user_ta() function from tee_ta_manager.c:
>
/* * Allocate heap and stack */ ctx->mm_heap_stack = tee_mm_alloc(&tee_mm_sec_ddr, *heap_size + ctx->stack_size); if (!ctx->mm_heap_stack) { EMSG("Failed to allocate %zu bytes\n", *heap_size + ctx->stack_size); EMSG(" of memory for user heap and stack\n"); return TEE_ERROR_OUT_OF_MEMORY; }
>
Is this allocating TA stack/heap? It seems to me that stack/heap is allocated from tee_mm_sec_ddr buffer defined in tee_mm_unpg.c:
>
/* Physical Secure DDR pool */ tee_mm_pool_t tee_mm_sec_ddr __data; /* XXX __data is a workaround */
>
To me this looks like all stacks are allocated from the same buffer. Perhaps I'm just looking at wrong place, or this is not the actual TA stack being allocated here... It seems I'm just missing something here. Can you please help clarify this?
OK, the memory is allocated from the same buffer but it's only what's allocated for a particular TA that is mapped in the VA space for that TA. Even if we only ask for a few kB with tee_mm_alloc(), the function will reserve the smallest mappable unit (currently 1 or 2 meg).
Regards, Jens
Thanks Jens!
I've briefly looked through tee_mm_alloc() and it does apply some shift which is 20 bit in case of TA DDR pool. So, there is no security concerns here. Only DDR size is a bit of an issue.
Thanks again, Sergey
-----Original Message----- From: Jens Wiklander [mailto:jens.wiklander@linaro.org] Sent: August-18-15 1:40 AM To: Blotsky, Sergey Cc: tee-dev@lists.linaro.org Subject: Re: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Hi,
On Mon, Aug 17, 2015 at 02:26:36PM +0000, Blotsky, Sergey wrote:
Hi Jens,
Thanks for your response, it does help to clarify our understanding of OP-TEE.
As for the stack/heap allocation, I was referring to the following code in tee_ta_load_user_ta() function from tee_ta_manager.c:
>
/* * Allocate heap and stack */ ctx->mm_heap_stack = tee_mm_alloc(&tee_mm_sec_ddr, *heap_size + ctx->stack_size); if (!ctx->mm_heap_stack) { EMSG("Failed to allocate %zu bytes\n", *heap_size + ctx->stack_size); EMSG(" of memory for user heap and stack\n"); return TEE_ERROR_OUT_OF_MEMORY; }
>
Is this allocating TA stack/heap? It seems to me that stack/heap is allocated from tee_mm_sec_ddr buffer defined in tee_mm_unpg.c:
>
/* Physical Secure DDR pool */ tee_mm_pool_t tee_mm_sec_ddr __data; /* XXX __data is a workaround */
>
To me this looks like all stacks are allocated from the same buffer. Perhaps I'm just looking at wrong place, or this is not the actual TA stack being allocated here... It seems I'm just missing something here. Can you please help clarify this?
OK, the memory is allocated from the same buffer but it's only what's allocated for a particular TA that is mapped in the VA space for that TA. Even if we only ask for a few kB with tee_mm_alloc(), the function will reserve the smallest mappable unit (currently 1 or 2 meg).
Regards, Jens
Jens,
Thanks again for your help.
I have couple of other questions: - does OP-TEE allow for scheduling TA threads periodically (as opposed to standard Global Platform entry points)? - does it allow for TAs having more than one thread? - I assume if there are multiple TAs that have pending tasks, then they will be scheduled in round-robin fashion, right?
Thanks, Sergey
-----Original Message----- From: Blotsky, Sergey Sent: August-18-15 10:35 AM To: 'Jens Wiklander' Cc: 'tee-dev@lists.linaro.org' Subject: RE: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Thanks Jens!
I've briefly looked through tee_mm_alloc() and it does apply some shift which is 20 bit in case of TA DDR pool. So, there is no security concerns here. Only DDR size is a bit of an issue.
Thanks again, Sergey
-----Original Message----- From: Jens Wiklander [mailto:jens.wiklander@linaro.org] Sent: August-18-15 1:40 AM To: Blotsky, Sergey Cc: tee-dev@lists.linaro.org Subject: Re: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Hi,
On Mon, Aug 17, 2015 at 02:26:36PM +0000, Blotsky, Sergey wrote:
Hi Jens,
Thanks for your response, it does help to clarify our understanding of OP-TEE.
As for the stack/heap allocation, I was referring to the following code in tee_ta_load_user_ta() function from tee_ta_manager.c:
>
/* * Allocate heap and stack */ ctx->mm_heap_stack = tee_mm_alloc(&tee_mm_sec_ddr, *heap_size + ctx->stack_size); if (!ctx->mm_heap_stack) { EMSG("Failed to allocate %zu bytes\n", *heap_size + ctx->stack_size); EMSG(" of memory for user heap and stack\n"); return TEE_ERROR_OUT_OF_MEMORY; }
>
Is this allocating TA stack/heap? It seems to me that stack/heap is allocated from tee_mm_sec_ddr buffer defined in tee_mm_unpg.c:
>
/* Physical Secure DDR pool */ tee_mm_pool_t tee_mm_sec_ddr __data; /* XXX __data is a workaround */
>
To me this looks like all stacks are allocated from the same buffer. Perhaps I'm just looking at wrong place, or this is not the actual TA stack being allocated here... It seems I'm just missing something here. Can you please help clarify this?
OK, the memory is allocated from the same buffer but it's only what's allocated for a particular TA that is mapped in the VA space for that TA. Even if we only ask for a few kB with tee_mm_alloc(), the function will reserve the smallest mappable unit (currently 1 or 2 meg).
Regards, Jens
On Tue, Aug 18, 2015 at 09:04:39PM +0000, Blotsky, Sergey wrote:
Jens,
Thanks again for your help.
I have couple of other questions:
- does OP-TEE allow for scheduling TA threads periodically (as opposed to standard Global Platform entry points)?
No, we don't have any design for that.
- does it allow for TAs having more than one thread?
We're following GP specifications so only one active thread per TA instance is allowed. Depending on how the TA is configured it will either have one instace for all sessions, or it will get a separate instance for each session. In the former case different sessions can share a state inside the TA while in the latter each session has its own state inside the TA.
- I assume if there are multiple TAs that have pending tasks, then they will be scheduled in round-robin fashion, right?
We don't have any policy in the code that enforces round-robin, but I assume that the natural order will be more or less round-robin.
Thanks, Sergey
-----Original Message----- From: Blotsky, Sergey Sent: August-18-15 10:35 AM To: 'Jens Wiklander' Cc: 'tee-dev@lists.linaro.org' Subject: RE: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Thanks Jens!
I've briefly looked through tee_mm_alloc() and it does apply some shift which is 20 bit in case of TA DDR pool. So, there is no security concerns here. Only DDR size is a bit of an issue.
Agree, we'll have to do something about that at some point.
Regards, Jens
Thanks a lot! Sergey
-----Original Message----- From: Jens Wiklander [mailto:jens.wiklander@linaro.org] Sent: August-19-15 1:53 AM To: Blotsky, Sergey Cc: tee-dev@lists.linaro.org Subject: Re: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
On Tue, Aug 18, 2015 at 09:04:39PM +0000, Blotsky, Sergey wrote:
Jens,
Thanks again for your help.
I have couple of other questions:
- does OP-TEE allow for scheduling TA threads periodically (as opposed to standard Global Platform entry points)?
No, we don't have any design for that.
- does it allow for TAs having more than one thread?
We're following GP specifications so only one active thread per TA instance is allowed. Depending on how the TA is configured it will either have one instace for all sessions, or it will get a separate instance for each session. In the former case different sessions can share a state inside the TA while in the latter each session has its own state inside the TA.
- I assume if there are multiple TAs that have pending tasks, then they will be scheduled in round-robin fashion, right?
We don't have any policy in the code that enforces round-robin, but I assume that the natural order will be more or less round-robin.
Thanks, Sergey
-----Original Message----- From: Blotsky, Sergey Sent: August-18-15 10:35 AM To: 'Jens Wiklander' Cc: 'tee-dev@lists.linaro.org' Subject: RE: [Tee-dev] Need help understanding high level design of OP-TEE Secure OS
Thanks Jens!
I've briefly looked through tee_mm_alloc() and it does apply some shift which is 20 bit in case of TA DDR pool. So, there is no security concerns here. Only DDR size is a bit of an issue.
Agree, we'll have to do something about that at some point.
Regards, Jens
-
Blotsky, Sergey -
Jens Wiklander