[LLVMdev] Custom lowering DYNAMIC_STACKALLOC

[Rafael Ávila de Espíndola]

On 11-06-17 10:31 AM, Sanjoy Das wrote:
> Hi!
>
> I'm a GSoC student this year, working on implementing split stacks on LLVM.
>
> TL;DR: I'm facing some problems trying to get LLVM to generate the code
> I want, please help me out if you can spare some time. It involves the
> SelectionDAG, MachinsInstr and liveness analysis portions.
>
>
>
> I'm currently trying to implement alloca correctly. It essentially boils
> down to checking if the current stack block has enough space to hold the
> alloca'ed block of memory. If yes, going the conventional  way (bumping
> the RSP); otherwise calling into a function that allocates the memory
> from the heap [1]. The stack pointer is not modified in the second case.
>
> I am trying to implement this by:
>
> a. Custom lowering DYNAMIC_ALLOCA in case segmented stacks are enabled.
>
> b. Creating a X86ISD::SEG_ALLOCA node in LowerDYNAMIC_STACKALLOC if
> segmented stacks are enabled. (Right now all LowerDYNAMIC_STACKALLOC on
> x86 does is check for Windows and lower the call to X86ISD::WIN_ALLOCA).
>
> c. Having EmitLoweredSegAlloca do the checks, (calling the external
> function if needed) and, in both the cases, write the pointer to the
> allocated memory to RAX. If the function is called nothing extra needs
> to be done, since the return value stays at RAX. If the stack pointer
> was changed, I do a (X86::MOV64rr, X86::RAX).addReg(X86::RSP).
>
> d. Setting the value of the node to RAX in LowerDYNAMIC_STACKALLOC,
> making the last part of the function effectively look like this:
>
>    // Reg is RAX or EAX, based on the subtarget
>    Chain = DAG.getNode(X86ISD::SEG_ALLOCA, dl, NodeTys, Chain, Flag);
>    Flag = Chain.getValue(1);
>    Chain = DAG.getCopyFromReg(Chain, dl, Reg, SPTy).getValue(1);
>
>    SDValue Ops1[2] = { Chain.getValue(0), Chain };
>    return DAG.getMergeValues(Ops1, 2, dl);
>
> Firstly, I would also like some feedback on this implementation in general.
>
> Secondly, the problem I'm facing: in the final assembly generated, the
> move instruction to RAX, in (c) is absent. I suspected this has
> something to do with the liveness analysis pass. With
> -debug-only=liveintervals, I see this
>
> 304L	%vreg5<def>  = COPY %RAX<kill>; GR64:%vreg5

Is SEG_ALLOCA marked as writing to RAX?

Is this code in github? It has been a long time since I looked at 
selection dags, but I could take a look.

btw, have you got -view-isel-dags (and the other view dags options) 
working? They are really handy for debugging this stuff.

Cheers,
Rafael