[PATCH 5.10 265/270] spi: spi-sun4i: fix early activation