[PATCH v2] usb: port: Don't try to peer unused USB ports based on location