[llvm-dev] JIT, LTO and @llvm.global_ctors: Looking for advise

[Benoit Belley via llvm-dev]

Thanks for the hindsight.

I am currently working on a patch/potential fix which introduces a new
Linker::ImportIntrinsicGlobalVariables flag. The patch includes a unit
test reproducing the problem. Hopefully, that will help getting more
feedback. 

Note that it might take a while before I am allowed to upload the patch
since I need approval from Autodesk Legal department.

Cheers,
Benoit

Benoit Belley
Sr Principal Developer
M&E-Product Development Group
 
MAIN +1 514 393 1616
DIRECT +1 438 448 6304
FAX +1 514 393 0110
 
Twitter <http://twitter.com/autodesk>
Facebook <https://www.facebook.com/Autodesk>
 
Autodesk, Inc.
10 Duke Street
Montreal, Quebec, Canada H3C 2L7
www.autodesk.com <http://www.autodesk.com/>
 




From:  David Blaikie <dblaikie at gmail.com>
Date:  mardi 20 juin 2017 à 11:12
To:  Benoit Belley <benoit.belley at autodesk.com>, llvm-dev
<llvm-dev at lists.llvm.org>, Lang Hames <lhames at gmail.com>, Teresa Johnson
<tejohnson at google.com>
Subject:  Re: [llvm-dev] JIT, LTO and @llvm.global_ctors: Looking for
advise


>+Lang (for JIT) & Teresa (for LTO/ThinLTO).
>
>Sounds like maybe the LinkOnlyNeeded got reused for a bit more than the
>original intent & maybe there should be more than one flag here - not
>sure.
>
>On Mon, Jun 19, 2017 at 9:16 AM Benoit Belley via llvm-dev
><llvm-dev at lists.llvm.org> wrote:
>
>
>Hi Everyone,
>
>We are looking for advise regarding the proper use of LTO in
>conjunction with just-in time generated code. Our usage scenario goes
>as follows.
>
>  1. Our front-end generates an LLVM module.
>
>  2. A small runtime support library is linked-in. The runtime
>     library is distributed as bitcode. It is generated using "clang++
>     -emit-llvm' and 'llvm-link'. This allows LTO to kick-in and
>     functions from the runtime support library become candidates for
>     inlining. This is the desired effect that we are looking for.
>
>  3. The resulting LLVM module is compiled and dynamically loaded. We
>     are currently using the MCJIT API, but are planning to move to ORC
>     very soon.
>
>Our LLVM module linking code goes roughly as follows:
>
>  Linker linker(jittedModule);
>  std::unique_ptr<llvm::Module> moduleToLink(
>    getLazyIRFileModule(bcFileName, error, context));
>  linker.linkInModule(std::move(module),
>                      Linker::LinkOnlyNeeded |
>                      Linker::InternalizeLinkedSymbol);
>
>Our issue is with the Linker::LinkOnlyNeeded flag. Using it has a huge
>positive impact on link and compilation time :-). But, it causes the
>@llvm.global_ctors and @llvm.global_dtors references from the
>linked-in modules to be discarded :-(. AFAICT, the Linker code assumes
>ThinLTO when the LinkOnlyNeeded flags is specified, and full-LTO
>otherwise.
>
>To resolve this, we have locally patched
>llvm/lib/Linker/LinkModules.cpp with:
>
>  bool ModuleLinker::run() {
>
>    // ....
>
>    if (shouldImportIntrinsicGlobalVariables()) {
>      auto addIntrinsicGlobalVariable = [ValuesToLink,
>                                         srcM](llvm::StringRef name) {
>        if (GlobalValue *GV = SrcM->getGlobalVariable(name)) {
>          ValuesToLink.insert(GV);
>        }
>      };
>
>      // These are added here, because they must not be internalized.
>      addIntrinsicGlobalVariable("llvm.used");
>      addIntrinsicGlobalVariable("llvm.compiler.used");
>      addIntrinsicGlobalVariable("llvm.global_ctors");
>      addIntrinsicGlobalVariable("llvm.global_dtors");
>    }
>
>    // ...
>
>  }
>
>Questions:
>
>   1. Is attempting to use llvm::Linker appropriate for our usage
>      pattern ? Should we directly use llvm::IRMover instead ?
>
>   2. Or, is our patch to ModuleLinker::run() the way to go ? Should
>      we submit back a patch along these lines ?
>
>   3. Other suggestions ?
>
>[Note] We are currently using LLVM 4.0.1-rc2.
>
>Cheers,
>Benoit
>
>
>
>Benoit Belley
>Sr Principal Developer
>M&E-Product Development Group
>
>MAIN +1 514 393 1616 <tel:(514)%20393-1616>
>DIRECT +1 438 448 6304 <tel:(438)%20448-6304>
>FAX +1 514 393 0110 <tel:(514)%20393-0110>
>
>Twitter <http://twitter.com/autodesk>
>Facebook <https://www.facebook.com/Autodesk>
>
>Autodesk, Inc.
>10 Duke Street
>Montreal, Quebec, Canada H3C 2L7
>www.autodesk.com <http://www.autodesk.com> <http://www.autodesk.com/>
>
>
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev