spi: stm32-qspi: Fix kernel oops when unbinding driver