[PATCH 5.1 09/30] USB: serial: f81232: fix interrupt worker not stop