[PATCH 6.1 320/325] spi: spi-sun4i: fix early activation