[PATCH] D11437: Correct x86_64 fp128 calling convention

[Reid Kleckner via cfe-commits]

rnk added a comment.

In http://reviews.llvm.org/D11437#211165, @chh wrote:

> I tried to make X86_64ABIInfo::classify to return (Lo=SSE, Hi=NoClass) for fp128 long double type, or (Lo=SSE, Hi=SSEUp). That is not enough, although making fp128 Complex type to "Memory" worked.
>
> X86_64ABIInfo::classifyArgumentType and classifyReturnType
>  will classify fp128 type as "double" through the help of GetSSETypeAtOffset.
>  These two or three functions still need more changes to handle fp128.
>  So I used the special cases for fp128, which seemed simpler with lower risk.
>
> The mapping to register classes is quite complicated to decide converted parameter or return types. Although AMD64 spec has lengthy rules written this way, the rules are quite difficult to understand the mapping of fp128 type.
>
> Is there other way to simplify these classification functions?


I think the right approach is to classify as SSE+SSEUp. It didn't work for you because GetByteVectorType was turning fp128 types into <2 x double>, which will correctly use XMM registers, but is not the IR you wanted.

I have a patch that fixes the TODOs and simplifies the code, do you mind if I land that?


http://reviews.llvm.org/D11437