[llvm-dev] LLD status update and performance chart

[Rui Ueyama via llvm-dev]

On Tue, Dec 13, 2016 at 8:03 PM, Sean Silva <chisophugis at gmail.com> wrote:

>
>
> On Tue, Dec 13, 2016 at 11:02 AM, Mehdi Amini <mehdi.amini at apple.com>
> wrote:
>
>>
>> On Dec 13, 2016, at 10:06 AM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Tue, Dec 13, 2016 at 9:28 AM, Mehdi Amini <mehdi.amini at apple.com>
>> wrote:
>>
>>>
>>> > On Dec 13, 2016, at 5:55 AM, Rafael Avila de Espindola via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>> >
>>> > Sean Silva via llvm-dev <llvm-dev at lists.llvm.org> writes:
>>> >> This will also greatly facilitate certain measurements I'd like to do
>>> >> w.r.t. different strategies for avoiding memory costs for input files
>>> (esp.
>>> >> minor faults and dTLB costs). I've almost gotten to the point of
>>> >> implementing this just to do those measurements.
>>> >
>>> > If you do please keep it local. The bare minimum we have of library
>>> > support is already disproportionately painful and prevents easier
>>> sharing
>>> > with COFF. We should really not add more until the linker is done.
>>>
>>> This is so much in contrast with the LLVM development, I find it quite
>>> hard to see this as an acceptable position on llvm-dev.
>>>
>>
>> LLD is a subproject of the LLVM project, but as a product, LLD itself is
>> not LLVM nor Clang, so some technical decisions that make sense to them are
>> not directly be applicable or even inappropriate. As a person who spent
>> almost two years on the old LLD and 1.5 years on the new LLD, I can say
>> that Rafael's stance on focusing on making a good linker first really makes
>> sense. I can easily imagine that if we didn't focus on that, we couldn't
>> make this much progress over the past 1.5 year and would be stagnated at a
>> very basic level. Do you know if I'm a person who worked really hard on the
>> old (and probably "modular" whatever it means) linker so hard? I'm speaking
>> based on the experience. If you have an concrete idea how to construct a
>> linker from smaller modules,  please tell me. I still don't get what you
>> want. We can discuss concrete proposals, but "making it (more) modular" is
>> too vague and not really a proposal, so it cannot be a productive
>> discussion.
>>
>> That said, I think the current our "API" to allow users call our linker's
>> main function hit the sweet spot. I know at least a few LLVM-based language
>> developers who want to eliminate external dependencies and embed a linker
>> to their compilers. That's a reasonable usage, and I think allowing them to
>> pass a map from filename to MemoryBuffer objects makes sense, too. That
>> would be done without affecting the overall linker architecture. I don't
>> oppose to that idea, and if someone wrote a patch, I'm fine with that.
>>
>>
>> I’m totally willing to believe you that it is not possible to write the
>> fastest ELF linker on earth (or in the universe) with a library based and
>> reusable components approach. But clang is not the fastest C/C++ compiler
>> available, and LLVM is not the fastest compiler framework either!
>>
>> So as a project, it seems to me that LLVM has not put the tradeoff on the
>> speed/efficiency historically when it was to the detriment of
>> layering/component/modularity/reusability/…
>>
>> Writing the fastest linker possible is nice goal, I regret that a LLVM
>> subproject is putting this goal above layering/component/modularity/reusability/…
>> though.
>>
>
> I think you are misunderstanding. None of the things you object to are not
> done for performance reasons, but rather for reducing code complexity and
> making the code easier to read (believe it or not). In fact, Rui is usually
> the first one to reject performance improvements if they make the code
> harder to maintain.
>

Thank you for pointing this. Yes, reducing complexity and making code
easier to read are the most important goals. It didn't come to mind to
mention that because I took it for granted. I always care about that when I
create programs. Other "conscious" goals are secondary.

Last year, after struggling and spending a long period of time with the old
LLD, I finally realized one thing: a linker is not an easy program, but it
shouldn't be that hard to write. I started thinking day and night until all
parts of a linker work flawlessly in my head. During the thought
experiments I found a lot of things that looked entangled before were
actually orthogonal. Once I got everything, I wrote it down to code, and it
just worked with a very small amount of code. That's how the new linker was
born.

It was a happy surprise to know afterwards that the new linker was very
fast, but that's a byproduct of searching for a good design. By design, I
didn't mean layering/component/etc but algorithms and data structures. I
cannot emphasize enough about the their importance, particularly, data
structures. If you design good data structures, you can get everything such
as code readability or performance almost for free.

Extensibility is very important. If I did sacrifice good engineering
practices for speed or something, I would be the first victim of such
shortsighted thinking. Remember that almost all features we have in ld.bfd
or ld.gold were originally "new linker extensions" when they were
introduced. I tried hard to keep LLD as a playground that anyone can
experiment different ideas with. Otherwise we couldn't implement that many
features in the past 1.5 years. It seems that a majority of people who took
a look at the code are sort of in favor of it (or at least don't dislike it
that much).

So, I still don't get what Mehdi is complaining. Mehdi, I'm not baiting.
Seriously. I listened to you last time and made a change for you, though
you didn't even recognize it (that actually made me sad). If you have more
items on your wishlist, I'll listen to you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161214/25579426/attachment.html>