[PATCH] LTO: introduce object file-based on-disk module format.

[Peter Collingbourne]

On Fri, Jul 04, 2014 at 06:04:29PM -0400, Rafael Espíndola wrote:
> > The code now uses parseBitcodeFile instead of getLazyBitcodeModule, which
> > as far as I can tell causes the reference to the memory buffer in the
> > BitcodeReader to be released.
> 
> I see. So,  the area is a bit of  can of worms. Lets expands a bit:
> 
> * ld64 needs to read the bodies early (see bellow).
> * gold does not.
> * llvm-ar does not.
> 
> It is also extremely handy to layer LTOModule on top of IRObjectFile
> to avoid code duplication and make sure the symbol resolution in the
> linker stays consistent with what symbols are placed in the archive
> symbol table.
> 
> If we end up using IRObjectFile directly for the "first pass" in the
> gold plugin, then it is OK to keep reading the entire bitcode file in
> LTOModule. Otherwise we have to factor that so that we can avoid doing
> that from the gold plugin.  I *think* that using IRObjectFile for the
> first pass makes sense, but I will have to try it and benchmark it a
> bit to be sure.
> 
> Constructing MemoryBuffer from StringRef is also a bit odd. One option
> would be to try to make Binary and the IR reader never own the buffer
> and just take StringRef arguments. The only real case I know where
> that can be a problem is the C interface, but if that is really the
> only place we can create a wrapper that is only ever used for C.

SGTM.

> >> Can't you invert this? Put the ELF in
> >> the IR (or even represent .go_export directly in some metadata) and
> >> extent the gold plugin API to handle this?
> >
> > I decided not to use IR as the outer layer because I think that it would be
> > useful for external tools to be able to understand the contents of metadata
> > without needing to understand the ever changing bitcode format.
> >
> > In the specific case of Go, there exist libraries and tools such as [1]
> > which understand ELF and can read the metadata in the object files, and I'd
> > prefer not to have to teach those tools about bitcode.
> 
> BTW, gcc has both thin and fat files and a GO frontend. Do you know
> what do they do about the GO metadata when building with thin files in
> LTO?

It looks like thin LTO files are also object files, but they omit the object
code (similar to this proposal). So the object files contain only .go_export
and the GCC LTO sections.

> How is this represented when targeting OS X or windows? Do they still use ELF?

I believe they use the native object format, but I haven't checked. The Go
tools (which I added gccgo support for) currently only support ELF because
that was all that I needed, but they could easily support the other formats.

> >> Another advantage is that
> >> it should be easier to support fat COFF and MachO objects too.
> >
> > The patch is already testing that COFF and Mach-O work. Or did you mean
> > something else?
> 
> No, I had missed that it was looking for a ".llvmbc" section in all 3.
> 
> >> * If we do need to go with IR in ELF instead of ELF in IR, we also
> >> need to update lib/Object to handle it. We still want to get an
> >> IRObjectFile (by default at least) when given one of those files so
> >> that llvm-nm/llvm-ar do the right thing.
> >
> > Maybe all that is needed is a function that "interprets" SymbolicFiles,
> > i.e. that takes a SymbolicFile and returns back another SymbolicFile (which
> > would be an IRObjectFile in the IR-in-ELF case). This looks like it would be
> > sufficient for llvm-nm/llvm-ar. Later I guess we could extend this function
> > to return multiple SymbolicFiles if we want to pursue the IR-in-object ==
> > IR+object idea.
> 
> Maybe. It would still probably be nicer if the default construction
> functions (like createBinary) would find out the IR in section and use
> that.

OK, I think it's reasonable for those functions to unwrap the IR by default.
A client which needs the actual object file can always use createObjectFile.

Thanks,
-- 
Peter