[PATCH 5.11 092/342] soundwire: cadence: only prepare attached devices on clock stop