r238861 - [x86-64 ABI] Fix for PR23082: an assertion failure when passing/returning a wrapper union in a full YMM register.

[Andrea Di Biagio]

Author: adibiagio
Date: Tue Jun  2 14:34:40 2015
New Revision: 238861

URL: http://llvm.org/viewvc/llvm-project?rev=238861&view=rev
Log:
[x86-64 ABI] Fix for PR23082: an assertion failure when passing/returning a wrapper union in a full YMM register.

This patch fixes an assertion failure in method
'X86_64ABIInfo::GetByteVectorType'.

Method 'GetByteVectorType' (in TargetInfo.cpp) is responsible
for mapping a QualType 'Ty' (for an argument or return value) to an LLVM IR
type that, according to the ABI, must be passed in a XMM/YMM vector register.

When selecting the IR vector type, method 'GetByteVectorType' always tries to
choose the "best" IR vector type for the 'Ty' in input. In particular, if Ty
is a wrapper structure, it keeps unwrapping it until it finds a vector type VTy.
That VTy is the "preferred IR type".

However, function 'isSingleElementStructure' (used to unwrap structures) does
not know how to look through union types. So, before this patch, if Ty was in
a nest of wrapper structures with at least two union types, we would have
triggered an assertion failure (added at revision 230971).

With this patch, if method 'GetByteVectorType' fails to find the preferred
vector type, we just return a valid (although potentially 'less friendly')
vector type based on the type size. So, rather than asserting on an 'unexpected'
'Ty' in input, we conservatively return vector type <2 x double> if Ty is 16
bytes, or <4 x double> if Ty is 32 bytes.

Differential Revision: http://reviews.llvm.org/D10190

Modified:
    cfe/trunk/lib/CodeGen/TargetInfo.cpp
    cfe/trunk/test/CodeGenCXX/x86_64-arguments-avx.cpp

Modified: cfe/trunk/lib/CodeGen/TargetInfo.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CodeGen/TargetInfo.cpp?rev=238861&r1=238860&r2=238861&view=diff
==============================================================================
--- cfe/trunk/lib/CodeGen/TargetInfo.cpp (original)
+++ cfe/trunk/lib/CodeGen/TargetInfo.cpp Tue Jun  2 14:34:40 2015
@@ -2227,9 +2227,16 @@ llvm::Type *X86_64ABIInfo::GetByteVector
     Ty = QualType(InnerTy, 0);
 
   llvm::Type *IRType = CGT.ConvertType(Ty);
-  assert(isa<llvm::VectorType>(IRType) &&
-         "Trying to return a non-vector type in a vector register!");
-  return IRType;
+  if(isa<llvm::VectorType>(IRType))
+    return IRType;
+
+  // We couldn't find the preferred IR vector type for 'Ty'.
+  uint64_t Size = getContext().getTypeSize(Ty);
+  assert((Size == 128 || Size == 256) && "Invalid type found!");
+
+  // Return a LLVM IR vector type based on the size of 'Ty'.
+  return llvm::VectorType::get(llvm::Type::getDoubleTy(getVMContext()),
+                               Size / 64);
 }
 
 /// BitsContainNoUserData - Return true if the specified [start,end) bit range

Modified: cfe/trunk/test/CodeGenCXX/x86_64-arguments-avx.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CodeGenCXX/x86_64-arguments-avx.cpp?rev=238861&r1=238860&r2=238861&view=diff
==============================================================================
--- cfe/trunk/test/CodeGenCXX/x86_64-arguments-avx.cpp (original)
+++ cfe/trunk/test/CodeGenCXX/x86_64-arguments-avx.cpp Tue Jun  2 14:34:40 2015
@@ -13,3 +13,40 @@ PR22753 test(PR22753 x) {
   return x;
 }
 }
+
+namespace test2 {
+typedef double __m128d __attribute__((__vector_size__(16)));
+typedef float __m128 __attribute__((__vector_size__(16)));
+typedef double __m256d __attribute__((__vector_size__(32)));
+typedef float __m256 __attribute__((__vector_size__(32)));
+
+union U1 {
+  __m128  v1;
+  __m128d v2;
+};
+
+union UU1 {
+  union U1;
+  __m128d v3;
+};
+
+// CHECK: define <2 x double> @_ZN5test27PR23082ENS_3UU1E(<2 x double>
+UU1 PR23082(UU1 x) {
+  return x;
+}
+
+union U2 {
+  __m256  v1;
+  __m256d v2;
+};
+
+union UU2 {
+  union U2;
+  __m256d v3;
+};
+
+// CHECK: define <4 x double> @_ZN5test27PR23082ENS_3UU2E(<4 x double>
+UU2 PR23082(UU2 x) {
+  return x;
+}
+}