This series implements feature detection of hardware virtualization on Linux and macOS; the latter being my primary use case.
This yields approximately a 6x improvement using HVF on M3 Pro.
Signed-off-by: Tamir Duberstein tamird@gmail.com --- Changes in v2: - Use QEMU accelerator fallback (Alyssa Ross, Thomas Weißschuh). - Link to v1: https://lore.kernel.org/r/20241025-kunit-qemu-accel-macos-v1-0-2f30c26192d4@...
--- Tamir Duberstein (2): kunit: add fallback for os.sched_getaffinity kunit: enable hardware acceleration when available
tools/testing/kunit/kunit.py | 11 ++++++++++- tools/testing/kunit/kunit_kernel.py | 3 +++ tools/testing/kunit/qemu_configs/arm64.py | 2 +- 3 files changed, 14 insertions(+), 2 deletions(-) --- base-commit: 81983758430957d9a5cb3333fe324fd70cf63e7e change-id: 20241025-kunit-qemu-accel-macos-2840e4c2def5
Best regards,
Python 3.13 added os.process_cpu_count as a cross-platform alternative for the Linux-only os.sched_getaffinity. Use it when it's available and provide a fallback when it's not.
This allows kunit to run on macOS.
Signed-off-by: Tamir Duberstein tamird@gmail.com --- tools/testing/kunit/kunit.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index bc74088c458aee20b1a21fdeb9f3cb01ab20fec4..3a8cbb868ac559f68d047e38be92f7c64a3314ea 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -303,7 +303,16 @@ def massage_argv(argv: Sequence[str]) -> Sequence[str]: return list(map(massage_arg, argv))
def get_default_jobs() -> int: - return len(os.sched_getaffinity(0)) + if sys.version_info >= (3, 13): + if (ncpu := os.process_cpu_count()) is not None: + return ncpu + raise RuntimeError("os.process_cpu_count() returned None") + # See https://github.com/python/cpython/blob/b61fece/Lib/os.py#L1175-L1186. + if sys.platform != "darwin": + return len(os.sched_getaffinity(0)) + if (ncpu := os.cpu_count()) is not None: + return ncpu + raise RuntimeError("os.cpu_count() returned None")
def add_common_opts(parser: argparse.ArgumentParser) -> None: parser.add_argument('--build_dir',
On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein tamird@gmail.com wrote:
Python 3.13 added os.process_cpu_count as a cross-platform alternative for the Linux-only os.sched_getaffinity. Use it when it's available and provide a fallback when it's not.
This allows kunit to run on macOS.
Signed-off-by: Tamir Duberstein tamird@gmail.com
Looks plausible enough to me. Thanks very much!
Reviewed-by: David Gow davidgow@google.com
Cheers, -- David
tools/testing/kunit/kunit.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index bc74088c458aee20b1a21fdeb9f3cb01ab20fec4..3a8cbb868ac559f68d047e38be92f7c64a3314ea 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -303,7 +303,16 @@ def massage_argv(argv: Sequence[str]) -> Sequence[str]: return list(map(massage_arg, argv))
def get_default_jobs() -> int:
return len(os.sched_getaffinity(0))
if sys.version_info >= (3, 13):if (ncpu := os.process_cpu_count()) is not None:return ncpuraise RuntimeError("os.process_cpu_count() returned None")# See https://github.com/python/cpython/blob/b61fece/Lib/os.py#L1175-L1186.if sys.platform != "darwin":return len(os.sched_getaffinity(0))if (ncpu := os.cpu_count()) is not None:return ncpuraise RuntimeError("os.cpu_count() returned None")def add_common_opts(parser: argparse.ArgumentParser) -> None: parser.add_argument('--build_dir',
-- 2.47.0
On Tue, Nov 5, 2024 at 3:36 AM David Gow davidgow@google.com wrote:
On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein tamird@gmail.com wrote:
Python 3.13 added os.process_cpu_count as a cross-platform alternative for the Linux-only os.sched_getaffinity. Use it when it's available and provide a fallback when it's not.
This allows kunit to run on macOS.
Signed-off-by: Tamir Duberstein tamird@gmail.com
Looks plausible enough to me. Thanks very much!
Reviewed-by: David Gow davidgow@google.com
Cheers, -- David
Thanks David! While the next patch is still plausibly undergoing discussion, would it be possible to pick this one up? Without it kunit.py is not usable on macOS.
Use KVM or HVF if supported by the QEMU binary and available on the system.
This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
Before: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 10.145s
After: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 1.773s
Signed-off-by: Tamir Duberstein tamird@gmail.com --- tools/testing/kunit/kunit_kernel.py | 3 +++ tools/testing/kunit/qemu_configs/arm64.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-append', ' '.join(params + [self._kernel_command_line]), '-no-reboot', '-nographic', + '-accel', 'kvm', + '-accel', 'hvf', + '-accel', 'tcg', '-serial', self._serial] + self._extra_qemu_params # Note: shlex.join() does what we want, but requires python 3.8+. print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command)) diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644 --- a/tools/testing/kunit/qemu_configs/arm64.py +++ b/tools/testing/kunit/qemu_configs/arm64.py @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''', qemu_arch='aarch64', kernel_path='arch/arm64/boot/Image.gz', kernel_command_line='console=ttyAMA0', - extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on']) + extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein tamird@gmail.com wrote:
Use KVM or HVF if supported by the QEMU binary and available on the system.
This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
Before: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 10.145s
After: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 1.773s
Signed-off-by: Tamir Duberstein tamird@gmail.com
Thanks a lot.
I finally managed to dig up an arm64 machine and test this, and I can reproduce the performance improvement.
Reviewed-by: David Gow davidgow@google.com
Cheers, -- David
tools/testing/kunit/kunit_kernel.py | 3 +++ tools/testing/kunit/qemu_configs/arm64.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-append', ' '.join(params + [self._kernel_command_line]), '-no-reboot', '-nographic',
'-accel', 'kvm','-accel', 'hvf','-accel', 'tcg', '-serial', self._serial] + self._extra_qemu_params # Note: shlex.join() does what we want, but requires python 3.8+. print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644 --- a/tools/testing/kunit/qemu_configs/arm64.py +++ b/tools/testing/kunit/qemu_configs/arm64.py @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''', qemu_arch='aarch64', kernel_path='arch/arm64/boot/Image.gz', kernel_command_line='console=ttyAMA0',
extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])-- 2.47.0
On 02/11/2024 12:09, Tamir Duberstein wrote:
Use KVM or HVF if supported by the QEMU binary and available on the system.
This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
Before: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 10.145s
After: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 1.773s
Signed-off-by: Tamir Duberstein tamird@gmail.com
tools/testing/kunit/kunit_kernel.py | 3 +++ tools/testing/kunit/qemu_configs/arm64.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-append', ' '.join(params + [self._kernel_command_line]), '-no-reboot', '-nographic',
'-accel', 'kvm','-accel', 'hvf','-accel', 'tcg', '-serial', self._serial] + self._extra_qemu_paramsdiff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644 --- a/tools/testing/kunit/qemu_configs/arm64.py +++ b/tools/testing/kunit/qemu_configs/arm64.py @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''', qemu_arch='aarch64', kernel_path='arch/arm64/boot/Image.gz', kernel_command_line='console=ttyAMA0',
extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise performance regresses by about 20%.
Before this patch: ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux- [11:03:38] Elapsed time: 15.494s
After this patch: ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux- [11:10:47] Elapsed time: 19.099s
Thanks, Kristina
On Tue, Nov 5, 2024 at 8:36 AM Kristina Martsenko kristina.martsenko@arm.com wrote:
On 02/11/2024 12:09, Tamir Duberstein wrote:
Use KVM or HVF if supported by the QEMU binary and available on the system.
This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
Before: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 10.145s
After: ./tools/testing/kunit/kunit.py exec --arch arm64 [HH:MM:SS] Elapsed time: 1.773s
Signed-off-by: Tamir Duberstein tamird@gmail.com
tools/testing/kunit/kunit_kernel.py | 3 +++ tools/testing/kunit/qemu_configs/arm64.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-append', ' '.join(params + [self._kernel_command_line]), '-no-reboot', '-nographic',
'-accel', 'kvm','-accel', 'hvf','-accel', 'tcg', '-serial', self._serial] + self._extra_qemu_params # Note: shlex.join() does what we want, but requires python 3.8+. print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644 --- a/tools/testing/kunit/qemu_configs/arm64.py +++ b/tools/testing/kunit/qemu_configs/arm64.py @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''', qemu_arch='aarch64', kernel_path='arch/arm64/boot/Image.gz', kernel_command_line='console=ttyAMA0',
extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise performance regresses by about 20%.
Before this patch: ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux- [11:03:38] Elapsed time: 15.494s
After this patch: ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux- [11:10:47] Elapsed time: 19.099s
Thanks, Kristina
Hi Kristina, thanks for pointing that out. I'm able to reproduce the regression. I poked around and I can't find a way to enable `pauth-impdef` only for TCG, and the problem is that enabling it globally produces:
tools/testing/kunit/kunit.py exec --arch arm64 --make_options LLVM=1 --raw_output=all [15:34:05] Starting KUnit Kernel (1/1)... Running tests with: $ qemu-system-aarch64 -nodefaults -m 1024 -kernel .kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1 console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -accel kvm -accel hvf -accel tcg -serial stdio -machine virt -cpu max,pauth-impdef=on qemu-system-aarch64: -accel kvm: invalid accelerator kvm qemu-system-aarch64: falling back to HVF qemu-system-aarch64: can't apply global max-arm-cpu.pauth-impdef=on: Property 'max-arm-cpu.pauth-impdef' not found
This behavior is at least somewhat intentional[0]. I have filed a bug with qemu[1]. If someone can conceive of a way to achieve this, I'd be delighted to send a v3.
Link: https://gitlab.com/qemu-project/qemu/-/commit/92d6528dbb20c6aec4022dfd63c7ff... [0] Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]
On Tue, Nov 5, 2024 at 2:51 PM Tamir Duberstein tamird@gmail.com wrote:
This behavior is at least somewhat intentional[0]. I have filed a bug with qemu[1]. If someone can conceive of a way to achieve this, I'd be delighted to send a v3.
[cut]
Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]
There hasn't been movement on this matter here or on the upstream bug. We can either proceed with this patch as-is or restore the manual feature detection that was present in v1 (https://lore.kernel.org/all/20241025-kunit-qemu-accel-macos-v1-2-2f30c26192d...) in order to retain `pauth-impdef` for TCG. Could I get some guidance from maintainers please?
Tamir
linux-kselftest-mirror@lists.linaro.org
-
David Gow -
Kristina Martsenko -
Tamir Duberstein