[llvm-dev] problem with porting a custom calling convention to LLVM 11.0.1

[John Reppy via llvm-dev]

We are attempting to use LLVM as a backend for our Standard ML compiler and
have defined a custom calling convention that passes all arguments in
registers (details can be found in in http://dx.doi.org/10.4204/EPTCS.285.5).
We have this calling convention working in LLVM version 10.0.1 (and previous
versions), but when we ported it to 11.0.1, it fails on the most trivial of
examples.

I'm not sure if something changed in LLVM 11 that requires further changes
to support our calling convention, or if it is a bug in LLVM?

Here are the details.  The failure message is:

Can't get register for value!
UNREACHABLE executed at /Users/jhr/Work/smlnj/sml-llvm/llvm-codegen/llvm-11.0.1.src/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:1609!
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.	Program arguments: ../llvm-debug/bin/llc -O1 -o triv.s triv.ll 
1.	Running pass 'Function Pass Manager' on module 'triv.ll'.
2.	Running pass 'X86 DAG->DAG Instruction Selection' on function '@entry71'
0  llc                      0x00000001033fc73a llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 58
1  llc                      0x00000001033fccc9 PrintStackTraceSignalHandler(void*) + 25
2  llc                      0x00000001033fab9b llvm::sys::RunSignalHandlers() + 123
3  llc                      0x00000001033fea79 SignalHandler(int) + 249
4  libsystem_platform.dylib 0x00007fff692445fd _sigtramp + 29
5  libsystem_platform.dylib 000000000000000000 _sigtramp + 18446603338752178720
6  libsystem_c.dylib        0x00007fff6911a808 abort + 120
7  llc                      0x0000000103297854 llvm::llvm_unreachable_internal(char const*, char const*, unsigned int) + 180
8  llc                      0x0000000103028bb2 llvm::SelectionDAGBuilder::getValueImpl(llvm::Value const*) + 6642
9  llc                      0x000000010302714d llvm::SelectionDAGBuilder::getValue(llvm::Value const*) + 173
10 llc                      0x000000010301d92a llvm::SelectionDAGBuilder::visitBitCast(llvm::User const&) + 58
11 llc                      0x0000000103012d5b llvm::SelectionDAGBuilder::visit(unsigned int, llvm::User const&) + 1115
12 llc                      0x00000001030120dd llvm::SelectionDAGBuilder::visit(llvm::Instruction const&) + 141
13 llc                      0x0000000103127a4a llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, bool&) + 218
14 llc                      0x0000000103125b3d llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) + 6061
15 llc                      0x0000000103122501 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 2129
16 llc                      0x0000000100da6153 (anonymous namespace)::X86DAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 259
17 llc                      0x0000000101cf080d llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 541
18 llc                      0x00000001024d6dcc llvm::FPPassManager::runOnFunction(llvm::Function&) + 700
19 llc                      0x00000001024df035 llvm::FPPassManager::runOnModule(llvm::Module&) + 117
20 llc                      0x00000001024d7794 (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) + 772
21 llc                      0x00000001024d72b8 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 296
22 llc                      0x00000001024df3d1 llvm::legacy::PassManager::run(llvm::Module&) + 33
23 llc                      0x00000001007f3937 compileModule(char**, llvm::LLVMContext&) + 6663
24 llc                      0x00000001007f1807 main + 1447
25 libdyld.dylib            0x00007fff6904bcc9 start + 1
26 libdyld.dylib            0x0000000000000005 start + 18446603338754245437
Abort

and the IR immediately before the crash is

*** IR Dump After Safe Stack instrumentation pass ***
; Function Attrs: naked
define cc20 void @entry71(i64** %allocPtr, i64** %limitPtr, i64** %storePtr, i64* %0, i64* %1, i64* %2, i64* %3, i64* %4, i64* %5, i64* %6) #0 {
L_71:
  %7 = bitcast i64* %2 to void (i64**, i64**, i64**, i64*, i64*, i64*, i64*, i64*, i64*, i64*)*
  tail call cc20 void %7(i64** %allocPtr, i64** %limitPtr, i64** %storePtr, i64* undef, i64* undef, i64* %2, i64* %3, i64* %4, i64* %5, i64* %6)
  ret void
}

The Value in question is "%2" and is an "Argument".  We see this error for both the x86-64 and arm64 targets.

Using LLVM 10.0.1, where this example does not crash, the IR dump immediately following the above is:

# *** IR Dump After Finalize ISel and expand pseudo-instructions ***:
# Machine code for function entry71: IsSSA, TracksLiveness
Function Live Ins: $rdi in %0, $r14 in %1, $r15 in %2, $rsi in %5, $rbx in %6, $rcx in %7, $rdx in %8, $rbp in %9

bb.0.L_71:
  liveins: $rdi, $r14, $r15, $rsi, $rbx, $rcx, $rdx, $rbp
  %9:gr64 = COPY $rbp
  %8:gr64 = COPY $rdx
  %7:gr64 = COPY $rcx
  %6:gr64 = COPY $rbx
  %5:gr64_tc = COPY $rsi
  %2:gr64 = COPY $r15
  %1:gr64 = COPY $r14
  %0:gr64 = COPY $rdi
  $rdi = COPY %0:gr64
  $r14 = COPY %1:gr64
  $r15 = COPY %2:gr64
  %10:gr64 = IMPLICIT_DEF
  $r8 = COPY %10:gr64
  %11:gr64 = IMPLICIT_DEF
  $r9 = COPY %11:gr64
  $rsi = COPY %5:gr64_tc
  $rbx = COPY %6:gr64
  $rcx = COPY %7:gr64
  $rdx = COPY %8:gr64
  $rbp = COPY %9:gr64
  TCRETURNri64 %5:gr64_tc, 0, <regmask>, implicit $rsp, implicit $ssp, implicit $rdi, implicit $r14, implicit $r15, implicit $r8, implicit $r9, implicit $rsi, implicit $rbx, implicit $rcx, implicit $rdx, implicit $rbp

# End machine code for function entry71.


Any help in figuring this problem out will be appreciated.

	- John