[llvm-dev] LLD status update and performance chart

[Rui Ueyama via llvm-dev]

On Tue, Dec 13, 2016 at 1:33 PM, Hal Finkel <hfinkel at anl.gov> wrote:

>
>
> ------------------------------
>
> *From: *"Rui Ueyama" <ruiu at google.com>
> *To: *"Hal Finkel" <hfinkel at anl.gov>
> *Cc: *"Rafael Avila de Espindola" <rafael.espindola at gmail.com>,
> "llvm-dev" <llvm-dev at lists.llvm.org>, "Andrew Kelley" <
> superjoe30 at gmail.com>
> *Sent: *Tuesday, December 13, 2016 3:08:15 PM
>
> *Subject: *Re: [llvm-dev] LLD status update and performance chart
>
> On Tue, Dec 13, 2016 at 12:40 PM, Hal Finkel <hfinkel at anl.gov> wrote:
>
>> ----- Original Message -----
>> > From: "Rafael Avila de Espindola via llvm-dev" <llvm-dev at lists.llvm.org
>> >
>> > To: "Andrew Kelley" <superjoe30 at gmail.com>, "Rui Ueyama" <
>> ruiu at google.com>
>> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
>> > Sent: Tuesday, December 13, 2016 2:30:41 PM
>> > Subject: Re: [llvm-dev] LLD status update and performance chart
>> >
>> > Andrew Kelley via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> >
>> > > On Tue, Dec 13, 2016 at 1:06 PM, Rui Ueyama via llvm-dev <
>> > > llvm-dev at lists.llvm.org> wrote:
>> > >>
>> > >> That said, I think the current our "API" to allow users call our
>> > >> linker's
>> > >> main function hit the sweet spot. I know at least a few LLVM-based
>> > >> language
>> > >> developers who want to eliminate external dependencies and embed a
>> > >> linker
>> > >> to their compilers. That's a reasonable usage, and I think
>> > >> allowing them to
>> > >> pass a map from filename to MemoryBuffer objects makes sense, too.
>> > >> That
>> > >> would be done without affecting the overall linker architecture. I
>> > >> don't
>> > >> oppose to that idea, and if someone wrote a patch, I'm fine with
>> > >> that.
>> > >>
>> > >
>> > > As an LLVM-based language developer, this is exactly what I want to
>> > > do. In
>> > > short:
>> > >
>> > >  * Avoid depending on an external binary
>> > >  * Avoid a fork+exec
>> > >  * Avoid unnecessary use of the filesystem
>> > >
>> > > Does this feature compromise progress on the linker binary?
>> > >
>> > > Would it be reasonable for this feature to exist in the next minor
>> > > version
>> > > release of lld?
>> >
>> > I would please ask for it to wait. It requires designing how the
>> > linker
>> > script parser and the thin archive system will fetch new files as
>> > they
>> > are found.
>>
>> Instead of just waiting, I think we should discuss design requirements.
>> Clang, for example, has a VFS layer too (in include/clang/Basic/VirtualFileSystem.h)
>> which seems independent from Clang. Maybe we want to move it into LLVM and
>> reuse it from both Clang and LLD. Would this VFS layer meet the
>> requirements for an LLD VFS layer too?
>
>
> Sorry my negligence, but what does Clang VFS layer provide? Looking at the
> header file, it looks to me that it provides functionalities to hook all
> file operations, so that you can run Clang against, well, a virtual file
> system that may not exist as a OS-level file system at all, for example. Is
> my understanding correct?
>
> Yes, I believe that is correct.
>

I took a look a bit more closely at the VFS layer. I can imagine that a few
reasonable use cases of the abstraction. One is obviously what Andrew wants
-- in-memory linking. That would be useful for distributed build system
too. (You can virtualize a file system using fuse, but that's not
cross-platform.)

I don't remember who said that, but someone proposed me if LLD can support
something like a "linker-driven build system", in which objects are created
when a linker needs them. It is experimental so it wouldn't need a support
in the main trunk, but a virtual filesystem would allow such system.


>
> As to a JIT linker, I once wrote a small JIT C compiler which creates an
> object file in memory and then jump to the entry point of the file after
> relocating it in memory. From that experience (and my experience of LLD),
> I'd think JIT linker is much easier to write than static linkers, because
> JIT linkers don't need to interact with dynamic linkers. Static linkers
> have to create data that will be interpreted by the dynamic linker, and
> that's one of the most complicated things in LLD (the other is the linker
> script, but that's a different story.) So I don't expect that there are
> many things that we can share between JIT linkers and static linkers. Of
> course there may be something that I don't know well about JIT, but that's
> my feeling at this moment.
>
> Right now, both parts of LLVM have code to process relocations, produces
> PLTs, etc. Those are target-specific things and it would be nice not to
> have to code them twice. Obviously, I could be oversimplifying the
> situation.
>
>  -Hal
>
>
>
>>  -Hal
>>
>> >
>> > Also, please consider what is so bad about writing a file. If your
>> > language has anything like translation unites than most of the
>> > program
>> > should already be in .o files.
>> >
>> > Cheers,
>> > Rafael
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>>
>> --
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>
>
>
>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/69f17bee/attachment.html>