Hi,
As SHM DMA memory is not unregistered by tee_shm_free() Linux tee driver API in kexec path (Case A), we 're facing lot of issues including failures in OP-TEE xtest. Can any one of you suggest the solution.
*Problem Description*
Case A If tee_shm_free() is called in shutdown() of Linux tee_client_driver and kexec kernel is booted, shutdown() & tee_shm_free() are invoked but tee_shm_release() is not invoked on DMA SHM buffer.
Case B If tee_shm_free() is called on SHM DMA buffer previously allocated from rmmod path[module_exit()] It unregisters the SHM memory and sends OPTEE_MSG_CMD_UNREGISTER_SHM to optee_os through optee_shm_unregister().
*Call Sequence*
Case A: kexec path *.shutdown()-->tee_shm_free()-->dma_buf_put()*
Case B: rmmod path *.shutdown()-->tee_shm_free()-->tee_shm_release()-->optee_shm_unregister() -->optee_do_call_with_arg() [cmd = OPTEE_MSG_CMD_UNREGISTER_SHM]*
*Repercussions of the issue: xtest failure due to out of memory*
If we register a big buffer of say 8MB in Linux tee client driver, if the same memory is not unregistered, it can cause overhead of 2*1024*8 = 16KB memory overhead for shm page book keeping data structures calloc'd in optee_os. After kexec it causes 16 +16=32KB which significant memory on a minimal heap of size, say 64KB. This causes failures with asymmetric crypto operations of xtest due to out of memory error.
*Context* In Linux kernel tee_client_driver probe() we 're calling tee_shm_alloc() with flags=TEE_SHM_MAPPED | TEE_SHM_DMA_BUF
In remove() & shutdown() functions of the driver: we 're calling tee_shm_free() on shm reference allocated in probe.