[PATCH 5.4 88/91] Revert "usb: gadget: u_serial: Disable ep before setting port to null to fix the crash caused by port being null"