Hi,
Trying to port OP-TEE to Atmel SAMA5D2 board (ATSAMA5D2-XULT) - single core ARM A5. I do not see any serial messages after MMU gets enabled (bl cpu_mmu_enable), neither does op-tee go onto load linux kernel.
Intended boot flow: at91bootstrap -> u-boot (loads tee.bin/zImage/dtb) -> op-tee -> kernel
Things I have already verified: - UART is setup to use MEM_AREA_IO_NSEC and UART data is setup as __early_bss "static struct atmel_uart_data console_data __early_bss; register_phys_mem(MEM_AREA_IO_NSEC, CONSOLE_UART_BASE, CORE_MMU_DEVICE_SIZE);"
- UART is setup to use virtual address after MMU is enabled "static vaddr_t chip_to_base(struct serial_chip *chip) { struct atmel_uart_data *pd = container_of(chip, struct atmel_uart_data, chip); return io_pa_or_va(&pd->base); }
static void atmel_uart_putc(struct serial_chip *chip, int ch) { vaddr_t base = chip_to_base(chip); while (!(read32(base + ATMEL_UART_SR) & ATMEL_SR_TXRDY)) ; write32(ch, base + ATMEL_UART_THR); }"
- CFG_WITH_PAGER is not set - CFG_WITH_LPAE is not set - CFG_PL310 is not set (both in u-boot and in op-tee) - u-boot/kernel is setup to not use the DDR memory address range which op-tee uses (32MB space, 0x3E000000 to 0x40000000) - Issue seen with both master branch and 2.4.0 tag - Printing a message immediately after bl cpu_mmu_enable does not show up. "bl ugly_trace -> prints correctly bl cpu_mmu_enable bl ugly_trace -> does not print
void ugly_trace(void); void ugly_trace(void) { MSG("ugly trace"); }"
Log: DEBUG: [0x5f5ed35e] TEE-CORE:add_phys_mem:324: CFG_SHMEM_START type NSEC_SHM 0x3e000000 size 0x00400000 DEBUG: [0x5f5ed35e] TEE-CORE:add_phys_mem:324: CFG_TA_RAM_START type TA_RAM 0x3e500000 size 0x01b00000 DEBUG: [0x5f5ed35e] TEE-CORE:add_phys_mem:324: CFG_TEE_RAM_START type TEE_RAM 0x3e400000 size 0x00100000 DEBUG: [0x5f5ed35e] TEE-CORE:add_phys_mem:324: CONSOLE_UART_BASE type IO_NSEC 0xf8000000 size 0x00200000 DEBUG: [0x5f5ed35e] TEE-CORE:add_va_space:363: type RES_VASPACE size 0x00a00000 DEBUG: [0x5f5ed35e] TEE-CORE:dump_mmap_table:476: type TEE_RAM va 0x3e400000..0x3e4fffff pa 0x3e400000..0x3e4fffff size 0x00100000 (pgdir) DEBUG: [0x5f5ed35e] TEE-CORE:dump_mmap_table:476: type TA_RAM va 0x3e500000..0x3fffffff pa 0x3e500000..0x3fffffff size 0x01b00000 (pgdir) DEBUG: [0x5f5ed35e] TEE-CORE:dump_mmap_table:476: type NSEC_SHM va 0x40000000..0x403fffff pa 0x3e000000..0x3e3fffff size 0x00400000 (pgdir) DEBUG: [0x5f5ed35e] TEE-CORE:dump_mmap_table:476: type IO_NSEC va 0x40400000..0x405fffff pa 0xf8000000..0xf81fffff size 0x00200000 (pgdir) DEBUG: [0x5f5ed35e] TEE-CORE:dump_mmap_table:476: type RES_VASPACE va 0x40600000..0x40ffffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) MESSAGE: [0x5f5ed35e] TEE-CORE:core_init_mmu_regs:822: TTBR0 = 0x3e45404a MESSAGE: [0x5f5ed35e] TEE-CORE:core_init_mmu_regs:823: TTBR1 = 0x3e45404a MESSAGE: [0x5f5ed35e] TEE-CORE:ugly_trace:52: ugly trace
Any pointers on what to check next is appreciated.
Thanks, Akshay
Hello Akshay.
Trying to port OP-TEE to Atmel SAMA5D2 board (ATSAMA5D2-XULT) - single core ARM A5. I do not see any serial messages after MMU gets enabled (bl cpu_mmu_enable), neither does op-tee go onto load linux kernel.
Just to be sure: does CFG_TEE_LOAD_ADDR corresponds to the real address where u-boot loads optee image? Also, I'm bothered by thread id (0x5f5ed35e in your case). It should be 0x0.
Hi Volodymyr,
On Thu, May 4, 2017 at 4:41 PM, Volodymyr Babchuk vlad.babchuk@gmail.com wrote:
Just to be sure: does CFG_TEE_LOAD_ADDR corresponds to the real address where u-boot loads optee image?
CFG_TEE_LOAD_ADDR corresponds to the relocated address (real entry point) of optee.
//Build details: $ make PLATFORM=at91-sama5d2xult -j12 CFG_PAGEABLE_ADDR=0 CFG_NS_ENTRY_ADDR=0x22000000 CFG_DT_ADDR=0x21000000 CFG_DT=y DEBUG=y CFG_TEE_CORE_LOG_LEVEL=4 $ mkimage -A arm -O linux -C none -a 0x3E3fffe4 -e 0x3E400000 -d ./out/arm-plat-at91/core/tee.bin uTee Image Name: Created: Thu May 4 16:49:09 2017 Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 223092 Bytes = 217.86 kB = 0.21 MB Load Address: 3e3fffe4 Entry Point: 3e400000
//u-boot load details fatload mmc 1:1 0x21000000 at91-sama5d2_xplained.dtb; fatload mmc 1:1 0x22000000 zImage; fatload mmc 1:1 0x23000000 uTee; bootm 0x23000000 - 0x21000000 reading at91-sama5d2_xplained.dtb 31619 bytes read in 7 ms (4.3 MiB/s) reading zImage 3680520 bytes read in 211 ms (16.6 MiB/s) reading uTee 223156 bytes read in 17 ms (12.5 MiB/s) ## Booting kernel from Legacy Image at 23000000 ... Image Name: Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 223092 Bytes = 217.9 KiB Load Address: 3e3fffe4 Entry Point: 3e400000 Verifying Checksum ... OK ## Flattened Device Tree blob at 21000000 Booting using the fdt blob at 0x21000000 Loading Kernel Image ... OK Loading Device Tree to 3db5c000, end 3db66b82 ... OK
Starting kernel ...
DEBUG: TEE-CORE:add_phys_mem:324: CFG_SHMEM_START type NSEC_SHM 0x3e000000 size 0x00400000 DEBUG: TEE-CORE:add_phys_mem:324: CFG_TA_RAM_START type TA_RAM 0x3e500000 size 0x01b00000 DEBUG: TEE-CORE:add_phys_mem:324: CFG_TEE_RAM_START type TEE_RAM 0x3e400000 size 0x00100000 DEBUG: TEE-CORE:add_phys_mem:324: CONSOLE_UART_BASE type IO_NSEC 0xf8000000 size 0x00200000 DEBUG: TEE-CORE:add_va_space:363: type RES_VASPACE size 0x00a00000 DEBUG: TEE-CORE:dump_mmap_table:476: type TEE_RAM va 0x3e400000..0x3e4fffff pa 0x3e400000..0x3e4fffff size 0x00100000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type TA_RAM va 0x3e500000..0x3fffffff pa 0x3e500000..0x3fffffff size 0x01b00000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type NSEC_SHM va 0x40000000..0x403fffff pa 0x3e000000..0x3e3fffff size 0x00400000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type IO_NSEC va 0x40400000..0x405fffff pa 0xf8000000..0xf81fffff size 0x00200000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type RES_VASPACE va 0x40600000..0x40ffffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) MESSAGE: TEE-CORE:ugly_trace:52: ugly trace MESSAGE: TEE-CORE:core_init_mmu_regs:822: TTBR0 = 0x3e45804a MESSAGE: TEE-CORE:core_init_mmu_regs:823: TTBR1 = 0x3e45804a MESSAGE: TEE-CORE:ugly_trace:52: ugly trace
//Code #define DRAM0_BASE 0x20000000 #define DRAM0_SIZE (512 * 1024 * 1024) #define TZDRAM_BASE (DRAM0_BASE + DRAM0_SIZE - TZDRAM_SIZE) #define TZDRAM_SIZE (28 * 1024 * 1024) #define CFG_SHMEM_START (TZDRAM_BASE - CFG_SHMEM_SIZE) #define CFG_SHMEM_SIZE (4 * 1024 * 1024) #define CFG_TEE_RAM_START TZDRAM_BASE #define CFG_TEE_RAM_VA_SIZE (1024 * 1024) #define CFG_TEE_RAM_PH_SIZE CFG_TEE_RAM_VA_SIZE #ifndef CFG_TEE_LOAD_ADDR #define CFG_TEE_LOAD_ADDR CFG_TEE_RAM_START #endif #define CFG_TA_RAM_START ROUNDUP((TZDRAM_BASE + CFG_TEE_RAM_VA_SIZE), \ CORE_MMU_DEVICE_SIZE) #define CFG_TA_RAM_SIZE ROUNDDOWN((TZDRAM_SIZE - CFG_TEE_RAM_VA_SIZE), \ CORE_MMU_DEVICE_SIZE)
Also, I'm bothered by thread id (0x5f5ed35e in your case). It should be 0x0.
The weird thing is looking back at the console logs and looks like I have been getting different thread ids between reboots/builds!
//1st instance DEBUG: [0x525c875d] TEE-CORE: //2nd instance DEBUG: TEE-CORE: //3rd instance DEBUG: [0x5f5ed35e] TEE-CORE: //4th instance DEBUG: [0x2000] TEE-CORE:
Thanks for your quick response.
Hi Akshay,
Just to be sure: does CFG_TEE_LOAD_ADDR corresponds to the real address where u-boot loads optee image?
CFG_TEE_LOAD_ADDR corresponds to the relocated address (real entry point) of optee.
//Build details:
Hm, looks fine.
Also, I'm bothered by thread id (0x5f5ed35e in your case). It should be 0x0.
The weird thing is looking back at the console logs and looks like I have been getting different thread ids between reboots/builds!
Ah, I'm sorry. Thread local storage is being initialized later, so it is okay to see strange thread id's on early stages of boot.
Okay, so looks like there are some problem with MMU or MMU tables. You can define define DEBUG_XLAT_TABLE 1 on core_mmu_v7.c to take a look at the tables. They should be OK. But just to be sure... And one more question: are you sure that u-boot boots in Secure PL1 mode? Can you dump contents of SCR just to be sure?
Hi Volodymyr,
On Thu, May 4, 2017 at 5:52 PM, Volodymyr Babchuk vlad.babchuk@gmail.com wrote:
Okay, so looks like there are some problem with MMU or MMU tables. You can define define DEBUG_XLAT_TABLE 1 on core_mmu_v7.c to take a look at the tables. They should be OK. But just to be sure... And one more question: are you sure that u-boot boots in Secure PL1 mode? Can you dump contents of SCR just to be sure?
Below is a log with DEBUG_XLAT_TABLE set to 1, seems to look ok to me.
I also printed the SCR register from op-tee and it is set to 0x0.
I also modified the code to set AW bit to 1 in SCR register, the SCR register read back 0x20 but still resulted in the hang.
DEBUG: TEE-CORE:add_phys_mem:325: CFG_SHMEM_START type NSEC_SHM 0x3e000000 size 0x00400000 DEBUG: TEE-CORE:add_phys_mem:325: CFG_TA_RAM_START type TA_RAM 0x3e500000 size 0x01b00000 DEBUG: TEE-CORE:add_phys_mem:325: CFG_TEE_RAM_START type TEE_RAM 0x3e400000 size 0x00100000 DEBUG: TEE-CORE:add_phys_mem:325: CONSOLE_UART_BASE type IO_NSEC 0xf8000000 size 0x00200000 DEBUG: TEE-CORE:add_va_space:364: type RES_VASPACE size 0x00a00000 DEBUG: TEE-CORE:add_va_space:364: type SHM_VASPACE size 0x00a00000 DEBUG: TEE-CORE:dump_mmap_table:478: type TEE_RAM va 0x3e400000..0x3e4fffff pa 0x3e400000..0x3e4fffff size 0x00100000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:478: type TA_RAM va 0x3e500000..0x3fffffff pa 0x3e500000..0x3fffffff size 0x01b00000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:478: type NSEC_SHM va 0x40000000..0x403fffff pa 0x3e000000..0x3e3fffff size 0x00400000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:478: type IO_NSEC va 0x40400000..0x405fffff pa 0xf8000000..0xf81fffff size 0x00200000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:478: type RES_VASPACE va 0x40600000..0x40ffffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:478: type SHM_VASPACE va 0x41000000..0x419fffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) DEBUG: TEE-CORE: section map [3e400000 3e500000] MEM-RW-X-S DEBUG: TEE-CORE: section map [3e500000 40000000] MEM-RW-XN-S DEBUG: TEE-CORE: section map [40000000 40400000] MEM-RW-XN-NS DEBUG: TEE-CORE: section map [40400000 40600000] DEV-RW-XN-NS DEBUG: TEE-CORE: section map [40600000 41000000] not-mapped DEBUG: TEE-CORE: section map [41000000 41a00000] MEM-RW-XN-NS MESSAGE: TEE-CORE:print_scr:57: SCR: = 0x0 MESSAGE: TEE-CORE:ugly_trace:52: ugly trace MESSAGE: TEE-CORE:core_init_mmu_regs:830: TTBR0 = 0x3e45804a MESSAGE: TEE-CORE:core_init_mmu_regs:831: TTBR1 = 0x3e45804a MESSAGE: TEE-CORE:ugly_trace:52: ugly trace
Thanks for your continued feedback. Anything else I can check?
Hello Akshay,
Okay, so looks like there are some problem with MMU or MMU tables. You can define define DEBUG_XLAT_TABLE 1 on core_mmu_v7.c to take a look at the tables. They should be OK. But just to be sure... And one more question: are you sure that u-boot boots in Secure PL1 mode? Can you dump contents of SCR just to be sure?
Below is a log with DEBUG_XLAT_TABLE set to 1, seems to look ok to me.
Yeah, looks good.
Thanks for your continued feedback. Anything else I can check?
Hmm... Another obvious thing to check is CPSR.M. Apart of this I don't have any ideas. You also can check documentation on your SoC regarding TruztZone and security extensions. Maybe there are some special steps need to be taken.
Hardware debugger would definitely help there if you have access to one.
On Fri, May 5, 2017 at 3:21 PM, Volodymyr Babchuk vlad.babchuk@gmail.com wrote:
Hmm... Another obvious thing to check is CPSR.M. Apart of this I don't have any ideas.
Before the MMU is enabled it is: MESSAGE: TEE-CORE:print_scr:58: CPSR: = 0x600001d3
I just got a j-link debugger to work with the board. Here is a register dump when the board gets into the hung state. Looks like there is a abort, will try and dig deeper to narrow down the cause.
(gdb) mon halt target state: halted target halted in Thumb state due to debug-request, current mode: Abort cpsr: 0x600001f7 pc: 0x3e407538 MMU: enabled, D-Cache: enabled, I-Cache: enabled Data fault registers DFSR: 00001cc7, DFAR: 23fe6ed7 Instruction fault registers IFSR: 00001403, IFAR: 0779e1c9
(1890) r0 (/32): 0x800001D7 (dirty) (1891) r1 (/32): 0x00000000 (1892) r2 (/32): 0x00000AFF (1893) r3 (/32): 0x00000AFF (1894) r4 (/32): 0x00000000 (1895) r5 (/32): 0x00000000 (1896) r6 (/32): 0x00000000 (1897) r7 (/32): 0x00000000 (1898) r8 (/32): 0x00000000 (1899) r9 (/32): 0x3E45E450 (1900) r10 (/32): 0x000000DC (1901) r11 (/32): 0x3E000000 (1902) r12 (/32): 0xDFFD20D7 (1903) sp_usr (/32) (1904) lr_usr (/32) (1905) pc (/32): 0x3E407538 (1906) r8_fiq (/32) (1907) r9_fiq (/32) (1908) r10_fiq (/32) (1909) r11_fiq (/32) (1910) r12_fiq (/32) (1911) sp_fiq (/32) (1912) lr_fiq (/32) (1913) sp_irq (/32) (1914) lr_irq (/32) (1915) sp_svc (/32) (1916) lr_svc (/32) (1917) sp_abt (/32): 0x3E45ED90 (1918) lr_abt (/32): 0x3E41E259 (1919) sp_und (/32) (1920) lr_und (/32) (1921) cpsr (/32): 0x600001F7 (1922) spsr_fiq (/32) (1923) spsr_irq (/32) (1924) spsr_svc (/32) (1925) spsr_abt (/32) (1926) spsr_und (/32) (1927) sp (/32) (1928) lr (/32) (1929) sp_mon (/32) (1930) lr_mon (/32) (1931) spsr_mon (/32)
You also can check documentation on your SoC regarding TruztZone and security extensions. Maybe there are some special steps need to be taken.
Thanks, I will take a look at it.
Before the MMU is enabled it is: MESSAGE: TEE-CORE:print_scr:58: CPSR: = 0x600001d3
Also looks pretty good
I just got a j-link debugger to work with the board. Here is a register dump when the board gets into the hung state. Looks like there is a abort, will try and dig deeper to narrow down the cause.
Yes. Definitely abort.
(gdb) mon halt target state: halted target halted in Thumb state due to debug-request, current mode: Abort cpsr: 0x600001f7 pc: 0x3e407538 MMU: enabled, D-Cache: enabled, I-Cache: enabled Data fault registers DFSR: 00001cc7, DFAR: 23fe6ed7
Hm, there is very strange encoding in DFSR. If I'm not mistaken, FS == b10111 which is reserved value. And DFAR points to some strange address.
(1890) r0 (/32): 0x800001D7 (dirty) (1891) r1 (/32): 0x00000000 (1892) r2 (/32): 0x00000AFF (1893) r3 (/32): 0x00000AFF (1894) r4 (/32): 0x00000000 (1895) r5 (/32): 0x00000000 (1896) r6 (/32): 0x00000000 (1897) r7 (/32): 0x00000000 (1898) r8 (/32): 0x00000000 (1899) r9 (/32): 0x3E45E450 (1900) r10 (/32): 0x000000DC (1901) r11 (/32): 0x3E000000 (1902) r12 (/32): 0xDFFD20D7 (1903) sp_usr (/32) (1904) lr_usr (/32) (1905) pc (/32): 0x3E407538 (1906) r8_fiq (/32) (1907) r9_fiq (/32) (1908) r10_fiq (/32) (1909) r11_fiq (/32) (1910) r12_fiq (/32) (1911) sp_fiq (/32) (1912) lr_fiq (/32) (1913) sp_irq (/32) (1914) lr_irq (/32) (1915) sp_svc (/32) (1916) lr_svc (/32) (1917) sp_abt (/32): 0x3E45ED90 (1918) lr_abt (/32): 0x3E41E259
At least you know exact address of instruction which caused abort.
Hello Akshay,
Have you booted OP-TEE on your board?
I just came with another idea. Can you try to zero all OP-TEE memory in u-boot before loading OP-TEE image? I have found that .bss section being zeroed up in late boot stages. That can lead to many funny issues like yours, as C code expects that all global and static variables are initialized to 0.
On Mon, May 8, 2017 at 8:00 AM, Volodymyr Babchuk vlad.babchuk@gmail.com wrote:
Hello Akshay,
Have you booted OP-TEE on your board?
I havent been able to boot OP-TEE on the SAMA5D2 board yet (seems issue is either platform or Cortex A5 specific, since I have a i.MX6UL that boots fine with OP-TEE).
Currently I am seeing 2 issues (previous panic was resolved by using CFG_SECURE_TIME_SOURCE_REE as the time source instead of CFG_SECURE_TIME_SOURCE_CNTPCT. Issue 1: Serial console not working after MMU is enabled If I put a while(1) at the end of init_primary_helper then the code is stuck in the while(1) loop without any aborts. However I do not see any serial messages that should be printed after MMU gets enabled. If I try to pint the UART registers using the virtual address after halting the processor over JLink then it returns "00000002" which is incorrect data.
Serial log: DEBUG: TEE-CORE:add_phys_mem:324: CFG_SHMEM_START type NSEC_SHM 0x3e000000 size 0x00400000 DEBUG: TEE-CORE:add_phys_mem:324: CFG_TA_RAM_START type TA_RAM 0x3e500000 size 0x01b00000 DEBUG: TEE-CORE:add_phys_mem:324: CFG_TEE_RAM_START type TEE_RAM 0x3e400000 size 0x00100000 DEBUG: TEE-CORE:add_phys_mem:324: CONSOLE_UART_BASE type IO_NSEC 0xf8000000 size 0x00200000 DEBUG: TEE-CORE:add_va_space:363: type RES_VASPACE size 0x00a00000 DEBUG: TEE-CORE:dump_mmap_table:476: type TEE_RAM va 0x3e400000..0x3e4fffff pa 0x3e400000..0x3e4fffff size 0x00100000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type TA_RAM va 0x3e500000..0x3fffffff pa 0x3e500000..0x3fffffff size 0x01b00000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type NSEC_SHM va 0x40000000..0x403fffff pa 0x3e000000..0x3e3fffff size 0x00400000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type IO_NSEC va 0x40400000..0x405fffff pa 0xf8000000..0xf81fffff size 0x00200000 (pgdir) DEBUG: TEE-CORE:dump_mmap_table:476: type RES_VASPACE va 0x40600000..0x40ffffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) DEBUG: TEE-CORE: section map [3e400000 3e500000] MEM-RW-X-S DEBUG: TEE-CORE: section map [3e500000 40000000] MEM-RW-XN-S DEBUG: TEE-CORE: section map [40000000 40400000] MEM-RW-XN-NS DEBUG: TEE-CORE: section map [40400000 40600000] DEV-RW-XN-NS DEBUG: TEE-CORE: section map [40600000 41000000] not-mapped
Objdump <snip> 3e41e3f8: f7e7 fa02 bl 3e405800 <init_teecore> 3e41e3fc: 4603 mov r3, r0 3e41e3fe: 2b00 cmp r3, #0 3e41e400: d005 beq.n 3e41e40e <init_primary_helper+0x76> /mnt/data/irobot/tee/optee_os_master/core/arch/arm/kernel/generic_boot.c:611 3e41e402: 2300 movs r3, #0 3e41e404: 2200 movs r2, #0 3e41e406: 2100 movs r1, #0 3e41e408: 2000 movs r0, #0 3e41e40a: f7e8 fe23 bl 3e407054 <__do_panic> /mnt/data/irobot/tee/optee_os_master/core/arch/arm/kernel/generic_boot.c:612 3e41e40e: f649 5358 movw r3, #40280 ; 0x9d58 3e41e412: f6c3 6342 movt r3, #15938 ; 0x3e42 3e41e416: 9300 str r3, [sp, #0] 3e41e418: 2301 movs r3, #1 3e41e41a: 2203 movs r2, #3 3e41e41c: f44f 7119 mov.w r1, #612 ; 0x264 3e41e420: f241 30a3 movw r0, #5027 ; 0x13a3 3e41e424: f6c3 6043 movt r0, #15939 ; 0x3e43 3e41e428: f7ff fd32 bl 3e41de90 <trace_printf> /mnt/data/irobot/tee/optee_os_master/core/arch/arm/kernel/generic_boot.c:613 (discriminator 1) 3e41e42c: e7fe b.n 3e41e42c <init_primary_helper+0x94>
(gdb): mon halt target state: halted target halted in Thumb state due to debug-request, current mode: Supervisor cpsr: 0x600001f3 pc: 0x3e41e42c MMU: enabled, D-Cache: enabled, I-Cache: enabled (gdb) mon mdw 0x40420000 10 0x40420000: 00000002 00000002 00000002 00000002 00000002 00000002 00000002 00000002 0x40420020: 00000002 00000002
Issue2: Hang after jumping to Linux kernel? If I remove the above while(1) and let the board boot, it appears to be stuck in the address range (physical) of the Linux kernel. I am not sure if it needs to be using virtual addresses here... (gdb) mon halt target state: halted target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x40000093 pc: 0x22003314 MMU: enabled, D-Cache: enabled, I-Cache: enabled (gdb) mon reg
===== ARM registers (4536) r0 (/32): 0x00000055 (dirty) (4537) r1 (/32): 0xF8020000 (4538) r2 (/32): 0xF7020000 (4539) r3 (/32): 0x00000002 (4540) r4 (/32): 0x00000055 (4541) r5 (/32): 0x220042FD (4542) r6 (/32): 0x00000024 (4543) r7 (/32): 0x00000000 (4544) r8 (/32): 0x223B62F0 (4545) r9 (/32): 0x223A62F0 (4546) r10 (/32): 0x20008000 (4547) r11 (/32): 0x223A52CC (4548) r12 (/32): 0x00000020 (4549) sp_usr (/32) (4550) lr_usr (/32) (4551) pc (/32): 0x22003314 (4552) r8_fiq (/32) (4553) r9_fiq (/32) (4554) r10_fiq (/32) (4555) r11_fiq (/32) (4556) r12_fiq (/32) (4557) sp_fiq (/32) (4558) lr_fiq (/32) (4559) sp_irq (/32) (4560) lr_irq (/32) (4561) sp_svc (/32): 0x223A62C0 (4562) lr_svc (/32): 0x22000960 (4563) sp_abt (/32) (4564) lr_abt (/32) (4565) sp_und (/32) (4566) lr_und (/32) (4567) cpsr (/32): 0x40000093 (4568) spsr_fiq (/32) (4569) spsr_irq (/32) (4570) spsr_svc (/32) (4571) spsr_abt (/32) (4572) spsr_und (/32) (4573) sp (/32) (4574) lr (/32) (4575) sp_mon (/32) (4576) lr_mon (/32) (4577) spsr_mon (/32)
I just came with another idea. Can you try to zero all OP-TEE memory in u-boot before loading OP-TEE image? I have found that .bss section being zeroed up in late boot stages. That can lead to many funny issues like yours, as C code expects that all global and static variables are initialized to 0.
Good idea. I tried this but it did not resolve the above issues.
=> mw.w 0x3E000000 0x0 0x02000000
Thanks for your continued support :)
Hi Akshay,
Currently I am seeing 2 issues (previous panic was resolved by using CFG_SECURE_TIME_SOURCE_REE as the time source instead of CFG_SECURE_TIME_SOURCE_CNTPCT. Issue 1: Serial console not working after MMU is enabled If I put a while(1) at the end of init_primary_helper then the code is stuck in the while(1) loop without any aborts. However I do not see any serial messages that should be printed after MMU gets enabled. If I try to pint the UART registers using the virtual address after halting the processor over JLink then it returns "00000002" which is incorrect data.
Hmm, can you share your UART driver somewhere? I just want to take a look. Also, can JLink perform a table walk for a specified VA? If I remember well, it allows you to access to CP15...
Issue2: Hang after jumping to Linux kernel? If I remove the above while(1) and let the board boot, it appears to be stuck in the address range (physical) of the Linux kernel. I am not sure if it needs to be using virtual addresses here... (gdb) mon halt target state: halted target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x40000093 pc: 0x22003314
Have you checked what it code is at given pc?
Thanks for your continued support :)
You are welcome. I was in the same situation multiple times. It is good to have a fresh idea from someone else :)
Hi Volodymyr,
On Mon, May 8, 2017 at 2:22 PM, Volodymyr Babchuk vlad.babchuk@gmail.com wrote:
Hmm, can you share your UART driver somewhere? I just want to take a look. Also, can JLink perform a table walk for a specified VA? If I remember well, it allows you to access to CP15...
The UART issue was related to defining the UART memory space as MEM_AREA_IO_NSEC. If I change it to MEM_AREA_IO_SEC, then the messages are printed ok!
The UART1 range needs to be 0xF8020000 to 0xF8024000; however the MMU translation table sets up 1MB (0xf8000000 to 0xf80fffff).
Looking at SAMA5 datasheet the peripheral SECURAM falls under this range and the peripheral security type for it is "always secure". Could this be the reason why UART needs to be defined as MEM_AREA_IO_SEC? Does op-tee support 4k/16k mapping instead of 1MB to workaround this issue?
DEBUG: [0x400] TEE-CORE:add_phys_mem:324: CFG_SHMEM_START type NSEC_SHM 0x3e000000 size 0x00400000 DEBUG: [0x400] TEE-CORE:add_phys_mem:324: CFG_TA_RAM_START type TA_RAM 0x3e500000 size 0x01b00000 DEBUG: [0x400] TEE-CORE:add_phys_mem:324: CFG_TEE_RAM_START type TEE_RAM 0x3e400000 size 0x00100000 DEBUG: [0x400] TEE-CORE:add_phys_mem:324: CONSOLE_UART_BASE type IO_SEC 0xf8000000 size 0x00100000 DEBUG: [0x400] TEE-CORE:add_va_space:363: type RES_VASPACE size 0x00a00000 DEBUG: [0x400] TEE-CORE:dump_mmap_table:476: type TEE_RAM va 0x3e400000..0x3e4fffff pa 0x3e400000..0x3e4fffff size 0x00100000 (pgdir) DEBUG: [0x400] TEE-CORE:dump_mmap_table:476: type TA_RAM va 0x3e500000..0x3fffffff pa 0x3e500000..0x3fffffff size 0x01b00000 (pgdir) DEBUG: [0x400] TEE-CORE:dump_mmap_table:476: type NSEC_SHM va 0x40000000..0x403fffff pa 0x3e000000..0x3e3fffff size 0x00400000 (pgdir) DEBUG: [0x400] TEE-CORE:dump_mmap_table:476: type IO_SEC va 0x40400000..0x404fffff pa 0xf8000000..0xf80fffff size 0x00100000 (pgdir) DEBUG: [0x400] TEE-CORE:dump_mmap_table:476: type RES_VASPACE va 0x40500000..0x40efffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir) DEBUG: [0x400] TEE-CORE: section map [3e400000 3e500000] MEM-RW-X-S DEBUG: [0x400] TEE-CORE: section map [3e500000 40000000] MEM-RW-XN-S DEBUG: [0x400] TEE-CORE: section map [40000000 40400000] MEM-RW-XN-NS DEBUG: [0x400] TEE-CORE: section map [40400000 40500000] DEV-RW-XN-S DEBUG: [0x400] TEE-CORE: section map [40500000 40f00000] not-mapped MESSAGE: [0x400] TEE-CORE:core_init_mmu_regs:829: TTBR0 = 0x3e458059 MESSAGE: [0x400] TEE-CORE:core_init_mmu_regs:830: TTBR1 = 0x3e458059 INFO: TEE-CORE: DEBUG: [0x0] TEE-CORE:init_canaries:167: #Stack canaries for stack_tmp[0] with top at 0x3e45e7f8 DEBUG: [0x0] TEE-CORE:init_canaries:167: watch *0x3e45e7fc DEBUG: [0x0] TEE-CORE:init_canaries:168: #Stack canaries for stack_abt[0] with top at 0x3e45f038 DEBUG: [0x0] TEE-CORE:init_canaries:168: watch *0x3e45f03c DEBUG: [0x0] TEE-CORE:init_canaries:170: #Stack canaries for stack_thread[0] with top at 0x3e461078 DEBUG: [0x0] TEE-CORE:init_canaries:170: watch *0x3e46107c DEBUG: [0x0] TEE-CORE:init_canaries:170: #Stack canaries for stack_thread[1] with top at 0x3e4630b8 DEBUG: [0x0] TEE-CORE:init_canaries:170: watch *0x3e4630bc INFO: TEE-CORE: OP-TEE version: 2.4.0-68-g8f386ec-dev #1 Mon May 8 19:43:54 UTC 2017 arm INFO: TEE-CORE: Initialized DEBUG: [0x0] TEE-CORE:init_primary_helper:612: Primary CPU switching to normal world boot
Issue2: Hang after jumping to Linux kernel? If I remove the above while(1) and let the board boot, it appears to be stuck in the address range (physical) of the Linux kernel. I am not sure if it needs to be using virtual addresses here... (gdb) mon halt target state: halted target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x40000093 pc: 0x22003314
Have you checked what it code is at given pc?
I ended up disabling serial console in op-tee to prevent mapping 0xf8000000 to 0xf80fffff (1MB) as "secure" (to allow linux to access them). Even after doing that I still end up seeing the above issue (PC stuck at 0x22003314).
Disassembling shows code stuck checking TXEMPTY bit of UART_SR (presumably Linux earlyprintk) but the UART registers always returns incorrect data "00000002" (think its related to permission): (gdb) mon arm disassemble 0x22003310 0x22003310 0xe5913014 LDR r3, [r1, #0x14] 0x22003314 0xe3130c02 TST r3, #0x200 0x22003318 0x0afffffc BEQ 0x22003310 0x2200331c 0xe1a0f00e MOV r15, r14 0x22003320 0xf8020000 UNDEFINED INSTRUCTION 0x22003324 0xf7020000 UNDEFINED INSTRUCTION 0x22003328 0xe92d4030 STMDB r13!, {r4, r5, r14} 0x2200332c 0xe1a0e1a2 MOV r14, r2, LSR #0x3 0x22003330 0xe1a0c000 MOV r12, r0 0x22003334 0xe1a03001 MOV r3, r1
(5208) r0 (/32): 0x00000055 (dirty) (5209) r1 (/32): 0xF8020000 (5210) r2 (/32): 0xF7020000 (5211) r3 (/32): 0x00000002 (5212) r4 (/32): 0x00000055 (5213) r5 (/32): 0x220042FD (5214) r6 (/32): 0x00000024 (5215) r7 (/32): 0x00000000 (5216) r8 (/32): 0x223B62F0 (5217) r9 (/32): 0x223A62F0 (5218) r10 (/32): 0x20008000 (5219) r11 (/32): 0x223A52CC (5220) r12 (/32): 0x00000020 (5223) pc (/32): 0x22003314 (5233) sp_svc (/32): 0x223A62C0 (5234) lr_svc (/32): 0x22000960 (5239) cpsr (/32): 0x40000093
//Trying to view UART registers: (gdb) mon mdw 0xF8020000 8 0xf8020000: 00000002 00000002 00000002 00000002 00000002 00000002 00000002 00000002
Looking at the TTBR0/1 registers: (gdb) mon arm mrc 15 0 2 0 0 536887296 (0x20004000) (gdb) mon arm mrc 15 0 2 0 1 0
Looking at the translation table entries, looks like the UART memory region is being mapped as "secure": UART MSB bits (0xF80), entry offset in table (0xF80 * 4 = 0x3E00) Translation table entry is at: 0x20004000 + 0x3E00 = 0x20007e00 (gdb) mon mdw 0x20007e00 0x20007e00: f8000c12 NS bit is set to zero?
Questions: I noticed the TTBR0/1 register contents are different between OP-TEE and Linux. Is this expected? Which code is setting the TTBR0 register to 0x20004000 and its entries as 0xXXX00C12? Is this done by Linux, if so does it clear the NS bit by default? Are there any platform specific support that needs to be added for the hand-off between op-tee and linux to happen correctly?
Any guidance on this is appreciated, since I am unclear how the handoff between op-tee and Linux happens.
Thanks, Akshay
On Mon, May 8, 2017 at 6:17 PM, Akshay Bhat akshay.bhat@timesys.com wrote:
The UART issue was related to defining the UART memory space as MEM_AREA_IO_NSEC. If I change it to MEM_AREA_IO_SEC, then the messages are printed ok!
The UART1 range needs to be 0xF8020000 to 0xF8024000; however the MMU translation table sets up 1MB (0xf8000000 to 0xf80fffff).
Looking at SAMA5 datasheet the peripheral SECURAM falls under this range and the peripheral security type for it is "always secure". Could this be the reason why UART needs to be defined as MEM_AREA_IO_SEC? Does op-tee support 4k/16k mapping instead of 1MB to workaround this issue?
Closing the loop incase it helps anyone else: To resolve the above issue, SAMA5 Matrix needs be configured to allow non-secure access to UART (defaults to secure coming out of at91bootstrap).
Questions: I noticed the TTBR0/1 register contents are different between OP-TEE and Linux. Is this expected?
Yes this is expected, ARM has different copies of the above registers corresponding to secure vs. non-secure world.
Are there any platform specific support that needs to be added for the hand-off between op-tee and linux to happen correctly?
Bootloader needs to configure any memory/bus controllers (in this case SAMA5 matrix security registers) to allow access to memory/peripherals needed by normal world. Otherwise you will see weird memory errors and/or imprecise external aborts when Linux runs in normal world.
After the above changes am able to successfully run op-tee / Linux (L2 cache is still not working but that is for another day).
Thanks.
-
Akshay Bhat -
Volodymyr Babchuk