Problem Description

Case A
If tee_shm_free() is called in shutdown() of Linux tee_client_driver and kexec kernel is booted, shutdown() & tee_shm_free() are invoked but tee_shm_release() is not invoked on DMA SHM buffer.

Case B
If tee_shm_free() is called on SHM DMA buffer previously allocated from rmmod path[module_exit()]
It unregisters the SHM memory and sends OPTEE_MSG_CMD_UNREGISTER_SHM to optee_os through optee_shm_unregister().

Call Sequence

Case A: kexec path
.shutdown()-->tee_shm_free()-->dma_buf_put()

Case B: rmmod path
.shutdown()-->tee_shm_free()-->tee_shm_release()-->optee_shm_unregister() -->optee_do_call_with_arg() [cmd = OPTEE_MSG_CMD_UNREGISTER_SHM]

Repercussions of the issue: xtest failure due to out of memory

If we register a big buffer of say 8MB in Linux tee client driver, if the same memory is not unregistered, it can cause overhead of 210248 = 16KB memory overhead for shm page book keeping data structures calloc'd in optee_os. After kexec it causes 16 +16=32KB which significant memory on a minimal heap of size, say 64KB. This causes failures with asymmetric crypto operations of xtest due to out of memory error.

Context
In Linux kernel tee_client_driver probe() we 're calling tee_shm_alloc() with flags=TEE_SHM_MAPPED | TEE_SHM_DMA_BUF

In remove() & shutdown() functions of the driver: we 're calling tee_shm_free() on shm reference allocated in probe.