[llvm-dev] [RFC] Support embedding bitcodes in LLD with LTO

[Rui Ueyama via llvm-dev]

On Fri, Feb 1, 2019 at 3:03 AM Josef Eisl <josef.eisl at oracle.com> wrote:

> Thanks for sharing your thoughts!
>
> On 31/01/2019 20:04, Rui Ueyama wrote:
> > That feature is probably too specific to your project. Most projects
> > that use LTO are using LTO just because it generates better code. Your
> > project is special as your program itself can also interpret LLVM
> > bitcode, but that's not the case for most other programs.
>
> I see that the requirement somewhat specific. On the other hand, the
> same feature is for example supported on Darwin via the __bundle
> section, so I'd see it more as a feature parity measure than something
> that is only of use to our project. (My view might be biased, though ;)
>
> > I think the option that's closest to the one you are looking for is
> > `--plugin-opt=emit-llvm`. That option makes lld to make an output file
> > in the bitcode file format (so lld doesn't do LTO if the option is given
> > and instead writes a raw bitcode as an output). With that option, I
> > don't think it's too hard to embed bitcode file to your executable. Run
> > the linker twice, with and without `--plugin-opt=emit-llvm`, and embed
> > the generated bitcode file using objcopy.
>  >
>  > Does that work for you?
>  >
>
> Sure it does and I fully agree that it is currently possible to get to
> the result. Actually, there are many different ways to accomplish it
> using combinations of wllvm, gllvm, llvm-link, --plugin-opt=emit-llvm,
> objcopy, etc. We are currently using these but my feelings about the
> approaches are mixed. I see two downsides.
> First, they require modifications to the build scripts. We want to
> support a variety of different build systems and make it as easy as
> possible for users to compile their projects for Sulong. Running the
> linker twice is not easy to accomplish in general, especially if the
> source project is not under our control. However, adding a linker flag
> is simpler in most cases.
> Second, the approaches add new dependencies and their portability is
> limited. E.g. what about objcopy on Darwin? Anyway, as mentioned above,
> Darwin does support embedding linked bitcode, but then we have two
> distinct workflows for Linux and Darwin, which I think is not very user
> friendly.
>
> That all said, I understand that there is some hesitation of bringing
> new, non-mainstream features to a project that needs to be stable and
> maintainable. Anyhow, the prototype we are currently experimenting with
> did no require too much work. It reuses existing code from clang (but
> unrelated to clang), which we moved to a common place in llvm. The rest
> of the patch is mainly option handling. There is hardly any duplication
> of logic. I see it more of an addition to existing functionality (i.e.,
> to `--plugin-opt=emit-llvm`). All the pieces are already there.
>
> Whatever the outcome of this RFC is, I guess I need thank you and all
> other contributors for providing such a stable and mature platform that
> allows us to do these kind of changes easily. Thanks. :)
>

Sorry for the late reply. I'd like to get an opinion from broader audiences.

 - Does anyone think that it is useful to have a linker option to embed
LLVM bitcode to a final exectuable when doing LTO? (Currently, LLVM bitcode
is discarded once LTO is done.)
 - If so, could you explain your use case?


>
> > On Thu, Jan 31, 2019 at 4:18 AM Josef Eisl <josef.eisl at oracle.com
> > <mailto:josef.eisl at oracle.com>> wrote:
> >
> >     Thanks for your response!
> >
> >     On 30/01/2019 20:18, Rui Ueyama wrote:
> >      > Hi Josef,
> >      >
> >      > Let me clarify my understanding. Do you want to keep original
> >     bitcode
> >      > files in the output executable when doing LTO, so that the
> resulting
> >      > executable contains both compiled bitcode (which is in native
> >     machine
> >      > instructions) and original bitcode files?
> >
> >     Exactly! Kind of analogous to what `clang -fembed-bitcode -c` does,
> but
> >     for executables.
> >
> >      >
> >      > Did you try embedding bitcode files into existing ELF files using
> >      > objcopy or linker option `--format=binary`?
> >
> >     Yes, that is the alternative. However, having support in the linker
> for
> >     that would require less tweaking of exiting build systems. Adding an
> >     option to CFLAGS/LDFLAGS would then be sufficient.
> >
> >      >
> >      > On Mon, Jan 28, 2019 at 6:41 AM Josef Eisl via llvm-dev
> >      > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>
> >     wrote:
> >      >
> >      >     Hi everybody!
> >      >
> >      >     I'm Josef and I'm working at Oracle Labs on Sulong [1,2], the
> >     LLVM IR
> >      >     execution engine in GraalVM [3]. In addition to executing
> >     bare bitcode
> >      >     files, Sulong also accepts ELF files with embedded bitcode
> >     sections.
> >      >     Therefore, it would be great if LLD in (Full)LTO mode would
> >     support
> >      >     embedding bitcode sections to the resulting object file. Is
> that
> >      >     something that would be considered useful and worth
> contributing?
> >      >
> >      >     Thanks,
> >      >     Josef
> >      >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190207/0203a072/attachment.html>