[LLVMdev] LLD use cases and design meeting and discussion

[Shankar Easwaram]

David,

I started a discussion yesterday on lld / chunks for ELF as we need relocations to be read when reading the inputs especially for handling comdat.

The other way thus can be done is doing symbol resolution while reading which makes the linker less suitable for concurrent operations.

Let me know what you think?

Shankar Easwaran 



> On Jun 5, 2015, at 23:35, Davide Italiano <davide at freebsd.org> wrote:
> 
>> On Fri, Jun 5, 2015 at 9:25 PM, Chandler Carruth <chandlerc at google.com> wrote:
>> Greetings folks,
>> 
>> Brief summary:
>> - Most of the active and interested parties in LLD sat down to discuss where
>> its going
>> - Many of the design misunderstandings (on all sides) were clarified, and
>> should make their way into documentation soon
>> - The various contributors have constructive paths forward, and the patches
>> and details about that will flow to the list as usual
>> 
>> If that's all you have time for, don't worry, you'll see all the results of
>> this meeting repeated on the lists and in patches and code reviews just like
>> usual. The rest is for those curious and particularly interested, or to
>> serve as a record for those who attended.
>> 
>> 
>> 
>> Earlier today, a bunch of the folks who are, have been, or have expressed a
>> particular interest in contributing to LLD got together in person to try to
>> exchange a bunch of ideas more rapidly than you realistically can in email
>> and discuss where we were seeing LLD go, why, and how it should progress
>> there.
>> 
>> I'm writing this email to share with the entire community (as we're of
>> course aware not everyone could join in the meeting feasibly) the highlights
>> of the discussion, and the specific plans that various contributors to LLD
>> made based on the discussion. We want anyone to feel free to jump in,
>> disagree, etc on the list, and of course any actual changes will be
>> discussed on their own as usual. Note that this summary is from my own
>> memory and notes, but will in may cases be relaying things from the entire
>> group that met rather than any particular view of my own, and I've CC-ed
>> those attending so they can correct anywhere I mess up.
>> 
>> 
>> The first subject we discussed were the use cases and features that the
>> various parties had for LLD. Essentially, what do we need, what do we want,
>> and some background of why. I couldn't begin to capture all of it, but below
>> are a list of interesting and informative examples. While not all users or
>> all targets of LLD would necessarily want all of these features, these are
>> things that design decisions in LLD should bear in mind. For example, these
>> should help the design of LLD not paint itself into a corner (too much).
>> However, there is no priority or ranking here, nor any commitment that these
>> are definite things on anyone's roadmap.
>> 
>> Interested features / use cases:
>> - Optimizing the resulting binary
>>  - GC-ing dead code
>>  - Shrinking the resulting binary through packing or other tricks
>>  - Programming language derived or semantic optimizations of layout
>>  - Laying out code to more efficiently {execute,load,etc}
>>  - ICF
>>  - Optimizing calls or indirections between functions / libraries based on
>> link structure
>> - Bug finding and/or analysis features
>>  - ODR violation detection
>>  - Library layering checking
>>  - Semantic or programming language checking
>>  - Verification or security hardening
>> - Compatibility
>>  - Drop-in command-line compatibility with linkers on all platforms
>>  - Link-time performance parity with existing platform linkers
>>  - Support for existing ABI constructs, synthesizing binary components, etc
>>  - Support for existing extremely wide variance of symbol resolution
>> strategies on different platforms
>> - Ability to achieve extensions like these via a plugin
>> - Rigorous testing methodology of all platforms
>> - Significant link-time improvements on some platforms where link-times are
>> a major pain point
>> - Support changes to standard build / development workflow
>>  - Linker servers or persistent linking process
>>  - Embedded usage of linker within other application via APIs
>>  - Different build system integration points that traditional steps
>>  - Toolkit for building linking related functionality
>>  - Potentially re-usable logic from JIT contexts
>> - Potential to decouple input/output file formats for particular platform(s)
>> - Manipulation of debug info (IE, more than just relocations)
>> - Object files controlling linker behavior via embedded flags or other
>> mechanisms
>> 
>> A closely related discussion was around what the right division between LLD
>> and LLVM was. There was broad consensus that LLD should provide the
>> libraries that handle everything necessary for linking, but not useful in
>> other tools. Everything necessary for linking but that *is* useful in other
>> tools should sink into the LLVM core infrastructure. An example of this
>> today would be a write API for libObject.
>> 
>> There was some further discussion about compatibility and the degree of
>> compatibility needs of different users of LLD. Among those in the room,
>> there were both users who would need initial compatibility with a system
>> linker, but would not likely need long-term compatibility layers, and those
>> for whom very long term (many years) compatibility would be a critical
>> feature. Essentially, we need to support both scenarios.
>> 
>> 
>> The final topic of significant discussion centered around whether there
>> should be a single core linking model in LLD, shared between all the
>> platforms and/or targets. There was little disagreement about having a
>> common model being useful and beneficial, but we currently don't have a
>> single model that adequately addresses the kinds of use cases outlined above
>> (specifically, there is a tension between link-time performance and feature
>> support on all platforms).
>> 
>> While not ideal, there was general willingness for LLD to provide
>> infrastructure supporting and platform support using multiple different
>> linking models if that is necessary to support the features (both functional
>> and performance) desired. However, there was also strong interest in
>> continually looking for opportunities to share basic code and
>> infrastructure, and even to collapse to a single model if one proves
>> sufficient for the use cases.
>> 
>> 
>> With that, several participants volunteered to take on specific tasks that I
>> just want to mention here for completeness. The results of these will of
>> course float into the mailing lists as usual.
>> 
>> 1) I promised to write up this summary to the list as host. =] Done!
>> 
>> 2) I have asked some of the contributors to LLD to ensure that the most
>> relevant parts of the above are actually incorporated into the
>> documentation. That will of course have its own patch review for detailed
>> discussion.
>> 
>> 3) Rui plans to continue to clean up and complete his prototype COFF
>> targeting code, in particular to make sure that the infrastructure it uses
>> is factored suitably to be re-used to target another target (if useful) and
>> that it is factored into a library with reasonable APIs (common APIs where
>> we can define them, and platform specific where necessary) to form a
>> reusable toolkit.
>> 
>> 4) Lang is going to look into the specific linking model in Rui's prototype
>> to see whether it would actually be able to support the use cases and
>> features that Darwin needs (and gets today with the Atom model). If it does,
>> we already have a path for converging on a single model. If it doesn't, we
>> will need to work to design APIs and library boundaries in a sensible way to
>> support divergent linking models between target platforms.
>> 
>> 5) If #4 proves to work, then everyone seemed interested in working to
>> systematically port. If #4 doesn't work, everyone is interested in working
>> to refactor the current libraries and the prototype so that the interface
>> boundaries and such all make sense again.
>> 
>> 
>> I hope that I've captured everything, both so that particularly interested
>> parties can chime in on any of the points, and so that those attending can
>> refer back to this where necessary.
>> 
>> Thanks to everyone who took the considerable time (and in a remarkably busy
>> week for some!) to try to work through a lot of these issues.
>> -Chandler
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> It would have been nice to know in advance about this, but alas.
> FWIW, For the past week I've been working on a rewrite of the ELF
> backend to be chunk based.
> It's still in a preliminary stage but maybe we should coordinate to
> duplicate effort on this side.
> 
> -- 
> Davide
> 
> "There are no solved problems; there are only problems that are more
> or less solved" -- Henri Poincare