[LLVMdev] Function-at-a-time Processing

[John Criswell]

Dear David,

Dumb question: Would it be easier to write an LLVM tool that took the 
large IR file and divided it into several smaller IR files that could 
then be code-generated separately?

-- John T.


David Greene wrote:
> We process very large programs and it is not unusual for the IR for some
> compilation unit to exceed system memory.  With some hacking in LLVM 2.5
> I was able to coax LLVM to generate asm for each functioin as it was
> processed and then completely forget about it (i.e. delete it) and move
> on to the next function.
>
> This required a bit of hackery.  I had to create two pass managers, one
> for the module and one for each function.  I called doInitialization on
> the module pass manager to convince the AsmPrinter to dump out global
> information needed at the top of the asm file.
>
> Then I would run each function though the function pass manager as they
> were encountered.  I could construct, run and destroy the function pass
> manager for each function.
>
> Finally I would run the module pass manager and doFinalization on the
> module pass manager to dump out the stuff needed at the end of the asm
> file.
>
> This was not at all pretty but I could at least get it to work.
>
> I'm in the last stages of upgrading to LLVM 2.7 and I'm finding that the
> new MCStreamer system is making this sort of model much more difficult.
> MCContexts are somehow shared between the separate module and function
> pass managers in ways I don't understand.  I think that somehow the
> LLVMTargetMachine is shared between the pass managers.
>
> I think this sort of sharing is actually correct since we want the
> MCContext for the function pass manager to be the same as the one for
> the module pass manager.  That way globals, etc. have consistent
> representations in both.  Unfortunately, when the first function pass
> manager gets deleted, it destroys the MCContext alonmg with it.
> Actually, the AsmPrinter destroys the MCContext through a reference (!)
> which makes things all the more confusing.  When the module pass manager
> comes along and tries of access an MCSection, everything blows up.
>
> I had to re-add addAssemblyEmitter to LLVMTargetMachine to try to make
> this work.  It essentially takes the AsmPrinter creation bits from
> addPassesToEmitFile and packages them into a separate function.
>
> [As a side note, I don't understand how the two AsmPrinters (one for the
> module pass manager, one for the function pass manager) end up pointing
> to the same MCContext as LLVMTargetMachine::addPassesToEmitFile()
> creates a new MCContext to pass to Target::createAsmPrinter.  Insights
> welcome.  :)]
>
> Perhaps there is a better way of doing this that's officially supported.
> Is this model of function processing supported at all?  If not, has
> anyone thought about how we might support it?  I think this is a model
> that is not all that uncommon so I would hope we could figure out a
> reasonable solution.
>
> Thanks!
>
>                      -Dave
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>