[PATCH 4.20 213/352] mlxsw: spectrum: Properly cleanup LAG uppers when removing port from LAG