Hi Paul,
Having PR_FP_MODE_FRE (i.e. Config5.FRE) set without PR_FP_MODE_FR (i.e. Status.FR) is not supported as the lone purpose of Config5.FRE is to emulate Status.FR=0 handling on FPU hardware that has Status.FR=1 hardwired[1][2]. Also we do not handle this case elsewhere, and assume throughout our code that TIF_HYBRID_FPREGS and TIF_32BIT_FPREGS cannot be set both at once for a task, leading to inconsistent behaviour if this does happen.
Reviewing the code I think we should actually end up with FR=1 in this case, because neither __own_fpu() nor the FPU emulator depend on the value of TIF_32BIT_FPREGS if TIF_HYBRID_FPREGS is set. So it's not too awful & I don't see the kernel doing anything too crazy, but it definitely isn't what the user asked for.
However `arch_ptrace' does check TIF_32BIT_FPREGS and gets things wrong if TIF_HYBRID_FPREGS is also set. Which is actually how I discovered this issue.
Return unsuccessfully then from prctl(2) PR_SET_FP_MODE calls requesting PR_FP_MODE_FRE to be set with PR_FP_MODE_FR clear. This corresponds to modes allowed by `mips_set_personality_fp'.
Looks good to me:
Reviewed-by: Paul Burton paul.burton@mips.com
Thanks for your review.
Maciej