[clang] [clang][codegen] Fix ABI for HVA/HFA returns on x86_64 MSVC (PR #113104)

[via cfe-commits]

================
@@ -1110,6 +1110,10 @@ static bool isTrivialForMSVC(const CXXRecordDecl *RD, QualType Ty,
       isa<VectorType>(Base)) {
     return true;
   }
+  if (CGM.getTarget().getTriple().isX86() &&
+      CGM.getABIInfo().isHomogeneousAggregate(Ty, Base, NumElts)) {
----------------
kin4stat wrote:

You were right — the previous version also affected non-vectorcall calls. For example, a non-aggregate {float, float} returned from a __cdecl function on x86_64 was returned in RAX, while MSVC returns it indirectly. The check is now in classifyReturnType and only applies to X86_VectorCall.

While comparing against MSVC 19.51 (x86 and x64), I also found that under vectorcall MSVC returns HVAs in vector registers from instance methods too, while we always used sret there. The same check now covers that.

The old test used an x86_64-windows-gnu triple, which uses the Itanium C++ ABI, so it didn’t exercise this code at all. I replaced it with microsoft-abi-vectorcall-hva-return.cpp. It runs for both x86_64 and i686 windows-msvc and covers vectorcall vs. cdecl, HFAs and HVAs, a non-HVA, and instance and static methods.

One remaining difference, which I’d prefer to leave for a follow-up: on x64, MSVC also returns HVAs with a non-trivial copy constructor, or with a non-trivial destructor and a size over 8 bytes, in registers. Fixing that means ignoring canPassInRegisters() for these returns, which seems worth a separate discussion.

Also rebased on main and squashed.

Code changes assisted by Claude Code, reviewed and tested manually by me

https://github.com/llvm/llvm-project/pull/113104