[llvm-dev] Have LLD load a file through memory

[Fangrui Song via llvm-dev]

On 2021-08-12, Frank Winter via llvm-dev wrote:
>Hi.
>
>I'm trying to use LLD to get it's input file from a string in memory instead of a file.
>
>My use case is a JIT compiler (IR --> AMD GCN ISA) where calling the linker to make it a loadable object via system() is hitting a bug (somewhere in the system).
>
>Here's what I do:
>
>  1.  With the AMDGPU backend I write the output to a string
>  2.  I sneak in this string into LLD via an added access function.
>  3.  Since LLD invocation is only used in this form:
>ld.lld -shared k.o -o k.so
>I modified 'elf::readFile' to
>
>  std::string tmp = qdp_jit_get_object();   // this retrieves the sneaky string
>
>  llvm::outs() << "size   = " << tmp.size() << "\n";
>
>  std::unique_ptr<MemoryBuffer> mb = MemoryBuffer::getMemBuffer( StringRef( tmp.c_str() , tmp.length() ) );
>
>  MemoryBufferRef mbref = mb->getMemBufferRef();
>
>  return mbref;
>
>
>When I then call LLD via lld::elf::link, I get
>
>size   = 2192
>ld: error: :1: unknown directive:
>>>> A�E�A�E�@���?V@���?V�3��������}j<������� �J�0�ւ���j�
>>>> ^

Looks like the file you passed to lld/ELF is not an ELF file.
lld/ELF parsed it as a linker script and bailed out.

Note, llvm-project/lld isn't ready when used as a library.
https://lists.llvm.org/pipermail/llvm-dev/2021-June/151031.html ("RFC:
Revisiting LLD-as-a-library design") lists some issues.

* cl::ResetAllOptionOccurrences && ParseCommandLineOptions() are called. The cl::opt, cl::list, ... options are reset.
   If the other part of the process using LLVM as a library needs to call into LLVM, resetting options may be a problem.
* One fatal error or more than --error-limit (default: 20) errors will exit the process.
   With CrashRecoveryContext, the exit calls becomes a signal raise. There can be random resource leaks.
* Warnings/erros are printed to stderr, which may interfere with the other part of the process.

Also be wary of the default uses all cores. If you have parallel links, you may need to specify
--threads=N.

People sometimes find lld::elf::link easy so use it instead of spawning a
process, but there are quite a few caveats.

>This is how the string containing the object was created:
>
>   string str_obj;
>   llvm::raw_string_ostream rss(str_obj);
>   llvm::buffer_ostream bos(rss);
>
>
>   if (TargetMachine->addPassesToEmitFile(CodeGenPasses,
>  bos,
>  nullptr,
>  llvm::CodeGenFileType::CGFT_ObjectFile ))
>
>CodeGenPasses.run(*Mod);
>
>
>Is there anything wrong how I turn the string into a MemoryBuffer so that the link fails?
>
>Thanks,
>Frank
>
>

>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev