[LLVMdev] Proposal: support object file-based on-disk module format

[Peter Collingbourne]

Hi,

Over in [1] we've been discussing adding support in LTO for an object
file-based on-disk module format. Rafael suggested that I send a proposal
to this list; this is that proposal.

As motivation, consider a compiler that needs to store metadata in the
LTO object file that may need to be read by future compilation steps,
such as the "export data" used by some Go compilers [2]. Such metadata
might also need to be read by external tools which do not know about LLVM,
so a good choice of file format would be something relatively stable and
well understood, readable without depending on LLVM and compatible with the
non-LTO scenario. This lends itself to the platform's native object file
format being a good candidate for the outermost file format, such that the
metadata and IR are stored in separate sections.

The basic proposal is that as an alternative on-disk representation for IR, we
also support native object files (i.e. ELF/COFF/Mach-O) with a section named
'.llvmbc' containing the bitcode in the same format that we are using now,
and no other (allocatable) sections. The actual support needed in LLVM would
be limited to consumers, i.e. LTO infrastructure: linker plugins, llvm-ar,
llvm-nm etc. We would not necessarily need to teach other bitcode consumers
(e.g. llvm-dis) about this format or add any producers to the tree, but
it may be useful as a matter of convenience to do so.

We can also consider extending this format by generating code into the
object file, such as for functions which we believe at compile time to
be cold, or for all functions if we want the decisions to be made at link
time. This may be beneficial for C/C++ compilation as it may allow us to
parallelize/deduplicate the code generation work for at least some functions.

Thanks,
-- 
Peter

[1] http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20140630/224359.html
[2] http://golang.org/doc/install/gccgo#Imports