Hi,

As SHM DMA memory is not unregistered by tee_shm_free() Linux tee driver API in kexec path (Case A), we 're facing lot of issues including failures in OP-TEE xtest. Can any one of you suggest the solution.

*Problem Description*

Case A If tee_shm_free() is called in shutdown() of Linux tee_client_driver and kexec kernel is booted, shutdown() & tee_shm_free() are invoked but tee_shm_release() is not invoked on DMA SHM buffer.

Case B If tee_shm_free() is called on SHM DMA buffer previously allocated from rmmod path[module_exit()] It unregisters the SHM memory and sends OPTEE_MSG_CMD_UNREGISTER_SHM to optee_os through optee_shm_unregister().

*Call Sequence*

Case A: kexec path *.shutdown()-->tee_shm_free()-->dma_buf_put()*

Case B: rmmod path *.shutdown()-->tee_shm_free()-->tee_shm_release()-->optee_shm_unregister() -->optee_do_call_with_arg() [cmd = OPTEE_MSG_CMD_UNREGISTER_SHM]*

*Repercussions of the issue: xtest failure due to out of memory*

If we register a big buffer of say 8MB in Linux tee client driver, if the same memory is not unregistered, it can cause overhead of 2*1024*8 = 16KB memory overhead for shm page book keeping data structures calloc'd in optee_os. After kexec it causes 16 +16=32KB which significant memory on a minimal heap of size, say 64KB. This causes failures with asymmetric crypto operations of xtest due to out of memory error.

*Context* In Linux kernel tee_client_driver probe() we 're calling tee_shm_alloc() with flags=TEE_SHM_MAPPED | TEE_SHM_DMA_BUF

In remove() & shutdown() functions of the driver: we 're calling tee_shm_free() on shm reference allocated in probe.