[cfe-dev] X86_64 ABI - Non-determiistic coersion

[Nickolas Pohilets via cfe-dev]

In the case of the following enum, result of coersion depends on the order
of union elements, which IMHO, is counter intuitive.

typedef union {
void* INT[2];
long double X87_X87UP;
double SSE;
} InReg;

InReg in_reg(InReg x) { return x; }

typedef union {
long double X87_X87UP;
double SSE;
void* INT[2];
} InMem;

InMem in_mem(InMem x) { return x; }

In IR these functions have the following signatures and are binary
incompatible:
define { i64, i64 } @in_reg(i64 %x.coerce0, i64 %x.coerce1)
define void @in_mem(%union.InMem* noalias sret %agg.result, %union.InMem*
byval align 16 %x)

This does not conflict with ABI specification, because the later does not
specify order in which union elements should be visited.

Do you think this is a real problem that is worth fixing in the ABI?

It is possible to fix this in the ABI by:
a) formalizing order in which fields are visited
b) changing merging and post-processing rules to have a consistent result

The first option is backward-compatible, the second allows to achieve
better performance of argument passing (I'm not expert in this field, but I
assume there are some performance concerns behind the current argument
classification rules).

Below is proposed algorithm to perform merging with consistent result:

To end up with { MEMORY, MEMORY } in the end:

   - Add two more classes NO_SSE and NO_X87
   - Merging INT with X87, X87_UP, COMPLEX_X87 results in NO_SSE
   - Merging INT with SSE or SSEUP results in NO_X87
   - Merging NO_SSE with SSE results in MEMORY
   - Merging NO_X87 with X87, X87_UP or COMPLEX_X87 results in MEMORY
   - Merging NO_SSE with NO_X87 cannot occur, because these classes may
   appear only as a result of merging - no field may have a class of NO_???
   - During post-merge phase NO_SSE and NO_X87 are replaced with INT

To end up with { INT, INT } in the end:

   - Add one more class MAYBE_MEMORY
   - Merging X87/X87_UP/COMPLEX_X87 with SSE/SSEUP results in MAYBE_MEMORY
   - Merging MAYBE_MEMORY with INT results in INT
   - During post-merge MAYBE_MEMORY is replaced with MEMORY

Thanks,
Nickolas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160618/a6cae085/attachment.html>