[clang] [clang][CodeGen][MSVC] Return vector types from methods indirectly (PR #157365)

[Reid Kleckner via cfe-commits]

================
@@ -1168,15 +1168,23 @@ static bool isTrivialForMSVC(const CXXRecordDecl *RD, QualType Ty,
 }
 
 bool MicrosoftCXXABI::classifyReturnType(CGFunctionInfo &FI) const {
-  const CXXRecordDecl *RD = FI.getReturnType()->getAsCXXRecordDecl();
-  if (!RD)
-    return false;
-
-  bool isTrivialForABI = RD->canPassInRegisters() &&
-                         isTrivialForMSVC(RD, FI.getReturnType(), CGM);
-
-  // MSVC always returns structs indirectly from C++ instance methods.
-  bool isIndirectReturn = !isTrivialForABI || FI.isInstanceMethod();
+  bool isIndirectReturn = false;
+  if (const CXXRecordDecl *RD = FI.getReturnType()->getAsCXXRecordDecl()) {
+    bool isTrivialForABI = RD->canPassInRegisters() &&
+                           isTrivialForMSVC(RD, FI.getReturnType(), CGM);
+
+    // MSVC always returns structs indirectly from C++ instance methods.
+    isIndirectReturn = !isTrivialForABI || FI.isInstanceMethod();
+  } else if (isa<VectorType>(FI.getReturnType())) {
+    // On x86, MSVC seems to only return vector types indirectly from non-
+    // vectorcall C++ instance methods.
+    isIndirectReturn =
+        CGM.getTarget().getTriple().isX86() && FI.isInstanceMethod() &&
----------------
rnk wrote:

I think it is worth expanding this out into `if` blocks. I think you can put the ABI ver check into the `else if` condition above, and combine the vector call check with a size check for > 64 bits to be ABI compatible with `__vectorcall` `__m64`. Please add a test for that as well.

https://github.com/llvm/llvm-project/pull/157365