[PATCH AUTOSEL 5.10 14/17] scsi: hisi_sas: Set a port invalid only if there are no devices attached when refreshing port id