[llvm-dev] Options for custom CCState, CCAssignFn, and GlobalISel

[Alex Bradbury via llvm-dev]

This question came about through reviewing work from Leslie Zhai on GlobalISel
support for RISC-V, which also motivated me to revisit code which I've always
felt was a bit clunky.

Calling convention lowering in LLVM is typically handled by functions
conforming to the CCAssignFn typedef:

    typedef bool CCAssignFn(unsigned ValNo, MVT ValVT,
                            MVT LocVT, CCValAssign::LocInfo LocInfo,
                            ISD::ArgFlagsTy ArgFlags, CCState &State);

Notably, these functions are called after type legalisation so an argument/ret
has been split to legal value types. In some cases you want more information
than is available through this function interface, which leads to a number of
backends creating their own CCState subclass:

* MipsCCState: adds bool vectors OriginalArgWasF128, OriginalArgWasFloat,
OriginalArgWasFloatVector, OriginalRetWasFloatVector, CallOperandIsFixed. Also
a SpeciallCallingConv field. Provides its own implementation of
AnalyzeFormalArguments etc that fill these vectors.
* HexagonCCState: adds a single extra field - NumNamedVarArgParams.
* PPCCCState: adds an OriginalArgWasPPCF128 bool vector. Arguably reduces
boilerplate vs MipsCCState by just having PPCISelLowering call
PPCCCState::PreAnalyzeCallOperands directly.
* SystemZCCState: has bool vectors ArgIsFixed and ArgIsShortVector, works
similarly to MipsCCState or PPCCCState.

The above works, but it isn't directly usable in the current GlobalISel
implementation. Very sensibly, GISel tries to both reuse existing calling
convention implementations and to reduce duplicated code as much as possible.
To this end, CallLowering::handleAssignments will create a CCState and use
ValueHandler::assignArg to call a function of type CCAssignFn type.

I see a couple of options:
1) Creating a new virtual method in GISel CallLowering that creates and
initialises a CCState or custom subclass. Use that to support target-specific
CCStates.
2) Try to remove the need for custom CCState altogether. In
<https://reviews.llvm.org/D38894>, Shiva Chen adds an OrigVT field to
ISD::ArgFlagsTy which seems much cleaner. It's not immediately obvious to me
if the in-tree users that currently track Type rather than EVT could always
make do with an EVT instead. [Input welcome!].
  * Do any out-of-tree backends have custom calling convention code that
requires more information than original arg type and whether the argument is
fixed or not? In the RISC-V calling convention implementation I'd be happiest
if the calling convention function had access to SubtargetInfo and the
DataLayout, but can probably do without.

Does anyone have views on this, or insight into planned future developments of
calling convention handling in GlobalISel?

Thanks,

Alex