On Mon, Jan 23, 2023, Erdem Aktas wrote:
On Mon, Jan 23, 2023 at 10:53 AM Sean Christopherson
seanjc@google.com wrote:
On Mon, Jan 23, 2023, Maciej S. Szmigiero wrote:
On 23.01.2023 19:30, Erdem Aktas wrote:
On Fri, Jan 20, 2023 at 4:28 PM Sean Christopherson
seanjc@google.com wrote:
On Sat, Jan 21, 2023, Ackerley Tng wrote: > Some SSE instructions assume a 16-byte aligned stack, and GCC
compiles
> assuming the stack is aligned: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838. This
combination
> results in a #GP in guests. > > Adding this compiler flag will generate an alternate prologue
and
> epilogue to realign the runtime stack, which makes selftest
code
> slower and bigger, but this is okay since we do not need
selftest code
> to be extremely performant.
Huh, I had completely forgotten that this is why SSE is
problematic. I ran into
this with the base UPM selftests and just disabled SSE.
/facepalm.
We should figure out exactly what is causing a misaligned
stack. As you've noted,
the x86-64 ABI requires a 16-byte aligned RSP. Unless I'm
misreading vm_arch_vcpu_add(),
the starting stack should be page aligned, which means
something is causing the
stack to become unaligned at runtime. I'd rather hunt down
that something than
paper over it by having the compiler force realignment.
Is not it due to the 32bit execution part of the guest code at
boot
time. Any push/pop of 32bit registers might make it a 16-byte unaligned stack.
32-bit stack needs to be 16-byte aligned, too (at function call
boundaries) -
see [1] chapter 2.2.2 "The Stack Frame"
And this showing up in the non-TDX selftests rules that out as the
sole problem;
the selftests stuff 64-bit mode, i.e. don't have 32-bit boot code.
Thanks Maciej and Sean for the clarification. I was suspecting the hand-coded assembly part that we have for TDX tests but it being happening in the non-TDX selftests disproves it.
Not necessarily, it could be both. Goofs in the handcoded assembly and PEBKAC on my end :-)
I figured it out!
GCC assumes that the stack is 16-byte aligned **before** the call instruction. Since call pushes rip to the stack, GCC will compile code assuming that on entrance to the function, the stack is -8 from a 16-byte aligned address.
Since for TDs we do a ljmp to guest code, providing a function's address, the stack was not modified by a call instruction pushing rip to the stack, so the stack is 16-byte aligned when the guest code starts running, instead of 16-byte aligned -8 that GCC expects.
For VMs, we set rip to a function pointer, and the VM starts running with a 16-byte algined stack too.
To fix this, I propose that in vm_arch_vcpu_add(), we align the allocated stack address and then subtract 8 from that:
@@ -573,10 +573,13 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id, vcpu_init_cpuid(vcpu, kvm_get_supported_cpuid()); vcpu_setup(vm, vcpu);
+ stack_vaddr += (DEFAULT_STACK_PGS * getpagesize()); + stack_vaddr = ALIGN_DOWN(stack_vaddr, 16) - 8; + /* Setup guest general purpose registers */ vcpu_regs_get(vcpu, ®s); regs.rflags = regs.rflags | 0x2; - regs.rsp = stack_vaddr + (DEFAULT_STACK_PGS * getpagesize()); + regs.rsp = stack_vaddr; regs.rip = (unsigned long) guest_code; vcpu_regs_set(vcpu, ®s);