[PATCH 4.19 180/313] mlxsw: spectrum: Properly cleanup LAG uppers when removing port from LAG