[PATCH 5.10.y] arm64: Restrict pagetable teardown to avoid false warning