<div dir="ltr">So this document now covers from the design philosophy to the actual design details. Do you have anything you want me to add?</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 14, 2016 at 2:46 PM, Rafael Espíndola <span dir="ltr"><<a href="mailto:rafael.espindola@gmail.com" target="_blank">rafael.espindola@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks!<br>
<br>
On 11 March 2016 at 22:06, Rui Ueyama via llvm-commits<br>
<div class="HOEnZb"><div class="h5"><<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br>
> Author: ruiu<br>
> Date: Sat Mar 12 00:06:40 2016<br>
> New Revision: 263336<br>
><br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=263336&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=263336&view=rev</a><br>
> Log:<br>
> Update the documents of the new LLD.<br>
><br>
> This patch merges the documents for ELF and COFF into one<br>
> and puts it into docs directory.<br>
><br>
> Added:<br>
> lld/trunk/docs/AtomLLD.rst<br>
> - copied, changed from r263292, lld/trunk/docs/index.rst<br>
> lld/trunk/docs/NewLLD.rst<br>
> Modified:<br>
> lld/trunk/COFF/README.md<br>
> lld/trunk/ELF/README.md<br>
> lld/trunk/docs/index.rst<br>
><br>
> Modified: lld/trunk/COFF/README.md<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/README.md?rev=263336&r1=263335&r2=263336&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/README.md?rev=263336&r1=263335&r2=263336&view=diff</a><br>
> ==============================================================================<br>
> --- lld/trunk/COFF/README.md (original)<br>
> +++ lld/trunk/COFF/README.md Sat Mar 12 00:06:40 2016<br>
> @@ -1,265 +1 @@<br>
> -The PE/COFF Linker<br>
> -==================<br>
> -<br>
> -This directory contains a linker for Windows operating system.<br>
> -Because the fundamental design of this port is different from<br>
> -the other ports of LLD, this port is separated to this directory.<br>
> -<br>
> -The linker is command-line compatible with MSVC linker and is<br>
> -generally 2x faster than that. It can be used to link real-world<br>
> -programs such as LLD itself or Clang, or even web browsers which<br>
> -are probably the largest open-source programs for Windows.<br>
> -<br>
> -This document is also applicable to ELF linker because the linker<br>
> -shares the same design as this COFF linker.<br>
> -<br>
> -Overall Design<br>
> ---------------<br>
> -<br>
> -This is a list of important data types in this linker.<br>
> -<br>
> -* SymbolBody<br>
> -<br>
> - SymbolBody is a class for symbols. They may be created for symbols<br>
> - in object files or in archive file headers. The linker may create<br>
> - them out of nothing.<br>
> -<br>
> - There are mainly three types of SymbolBodies: Defined, Undefined, or<br>
> - Lazy. Defined symbols are for all symbols that are considered as<br>
> - "resolved", including real defined symbols, COMDAT symbols, common<br>
> - symbols, absolute symbols, linker-created symbols, etc. Undefined<br>
> - symbols are for undefined symbols, which need to be replaced by<br>
> - Defined symbols by the resolver. Lazy symbols represent symbols we<br>
> - found in archive file headers -- which can turn into Defined symbols<br>
> - if we read archieve members, but we haven't done that yet.<br>
> -<br>
> -* Symbol<br>
> -<br>
> - Symbol is a pointer to a SymbolBody. There's only one Symbol for<br>
> - each unique symbol name (this uniqueness is guaranteed by the symbol<br>
> - table). Because SymbolBodies are created for each file<br>
> - independently, there can be many SymbolBodies for the same<br>
> - name. Thus, the relationship between Symbols and SymbolBodies is 1:N.<br>
> -<br>
> - The resolver keeps the Symbol's pointer to always point to the "best"<br>
> - SymbolBody. Pointer mutation is the resolve operation in this<br>
> - linker.<br>
> -<br>
> - SymbolBodies have pointers to their Symbols. That means you can<br>
> - always find the best SymbolBody from any SymbolBody by following<br>
> - pointers twice. This structure makes it very easy to find<br>
> - replacements for symbols. For example, if you have an Undefined<br>
> - SymbolBody, you can find a Defined SymbolBody for that symbol just<br>
> - by going to its Symbol and then to SymbolBody, assuming the resolver<br>
> - have successfully resolved all undefined symbols.<br>
> -<br>
> -* Chunk<br>
> -<br>
> - Chunk represents a chunk of data that will occupy space in an<br>
> - output. Each regular section becomes a chunk.<br>
> - Chunks created for common or BSS symbols are not backed by sections.<br>
> - The linker may create chunks out of nothing to append additional<br>
> - data to an output.<br>
> -<br>
> - Chunks know about their size, how to copy their data to mmap'ed<br>
> - outputs, and how to apply relocations to them. Specifically,<br>
> - section-based chunks know how to read relocation tables and how to<br>
> - apply them.<br>
> -<br>
> -* SymbolTable<br>
> -<br>
> - SymbolTable is basically a hash table from strings to Symbols, with<br>
> - a logic to resolve symbol conflicts. It resolves conflicts by symbol<br>
> - type. For example, if we add Undefined and Defined symbols, the<br>
> - symbol table will keep the latter. If we add Defined and Lazy<br>
> - symbols, it will keep the former. If we add Lazy and Undefined, it<br>
> - will keep the former, but it will also trigger the Lazy symbol to<br>
> - load the archive member to actually resolve the symbol.<br>
> -<br>
> -* OutputSection<br>
> -<br>
> - OutputSection is a container of Chunks. A Chunk belongs to at most<br>
> - one OutputSection.<br>
> -<br>
> -There are mainly three actors in this linker.<br>
> -<br>
> -* InputFile<br>
> -<br>
> - InputFile is a superclass of file readers. We have a different<br>
> - subclass for each input file type, such as regular object file,<br>
> - archive file, etc. They are responsible for creating and owning<br>
> - SymbolBodies and Chunks.<br>
> -<br>
> -* Writer<br>
> -<br>
> - The writer is responsible for writing file headers and Chunks to a<br>
> - file. It creates OutputSections, put all Chunks into them, assign<br>
> - unique, non-overlapping addresses and file offsets to them, and then<br>
> - write them down to a file.<br>
> -<br>
> -* Driver<br>
> -<br>
> - The linking process is drived by the driver. The driver<br>
> -<br>
> - - processes command line options,<br>
> - - creates a symbol table,<br>
> - - creates an InputFile for each input file and put all symbols in it<br>
> - into the symbol table,<br>
> - - checks if there's no remaining undefined symbols,<br>
> - - creates a writer,<br>
> - - and passes the symbol table to the writer to write the result to a<br>
> - file.<br>
> -<br>
> -Performance<br>
> ------------<br>
> -<br>
> -It's generally 2x faster than MSVC link.exe. It takes 3.5 seconds to<br>
> -self-host on my Xeon 2580 machine. MSVC linker takes 7.0 seconds to<br>
> -link the same executable. The resulting output is 65MB.<br>
> -The old LLD is buggy that it produces 120MB executable for some reason,<br>
> -and it takes 30 seconds to do that.<br>
> -<br>
> -We believe the performance difference comes from simplification and<br>
> -optimizations we made to the new port. Notable differences are listed<br>
> -below.<br>
> -<br>
> -* Reduced number of relocation table reads<br>
> -<br>
> - In the old design, relocation tables are read from beginning to<br>
> - construct graphs because they consist of graph edges. In the new<br>
> - design, they are not read until we actually apply relocations.<br>
> -<br>
> - This simplification has two benefits. One is that we don't create<br>
> - additional objects for relocations but instead consume relocation<br>
> - tables directly. The other is that it reduces number of relocation<br>
> - entries we have to read, because we won't read relocations for<br>
> - dead-stripped COMDAT sections. Large C++ programs tend to consist of<br>
> - lots of COMDAT sections. In the old design, the time to process<br>
> - relocation table is linear to size of input. In this new model, it's<br>
> - linear to size of output.<br>
> -<br>
> -* Reduced number of symbol table lookup<br>
> -<br>
> - Symbol table lookup can be a heavy operation because number of<br>
> - symbols can be very large and each symbol name can be very long<br>
> - (think of C++ mangled symbols -- time to compute a hash value for a<br>
> - string is linear to the length.)<br>
> -<br>
> - We look up the symbol table exactly only once for each symbol in the<br>
> - new design. This is I believe the minimum possible number. This is<br>
> - achieved by the separation of Symbol and SymbolBody. Once you get a<br>
> - pointer to a Symbol by looking up the symbol table, you can always<br>
> - get the latest symbol resolution result by just dereferencing a<br>
> - pointer. (I'm not sure if the idea is new to the linker. At least,<br>
> - all other linkers I've investigated so far seem to look up hash<br>
> - tables or sets more than once for each new symbol, but I may be<br>
> - wrong.)<br>
> -<br>
> -* Reduced number of file visits<br>
> -<br>
> - The symbol table implements the Windows linker semantics. We treat<br>
> - the symbol table as a bucket of all known symbols, including symbols<br>
> - in archive file headers. We put all symbols into one bucket as we<br>
> - visit new files. That means we visit each file only once.<br>
> -<br>
> - This is different from the Unix linker semantics, in which we only<br>
> - keep undefined symbols and visit each file one by one until we<br>
> - resolve all undefined symbols. In the Unix model, we have to visit<br>
> - archive files many times if there are circular dependencies between<br>
> - archives.<br>
> -<br>
> -* Avoiding creating additional objects or copying data<br>
> -<br>
> - The data structures described in the previous section are all thin<br>
> - wrappers for classes that LLVM libObject provides. We avoid copying<br>
> - data from libObject's objects to our objects. We read much less data<br>
> - than before. For example, we don't read symbol values until we apply<br>
> - relocations because these values are not relevant to symbol<br>
> - resolution. Again, COMDAT symbols may be discarded during symbol<br>
> - resolution, so reading their attributes too early could result in a<br>
> - waste. We use underlying objects directly where doing so makes<br>
> - sense.<br>
> -<br>
> -Parallelism<br>
> ------------<br>
> -<br>
> -The abovementioned data structures are also chosen with<br>
> -multi-threading in mind. It should relatively be easy to make the<br>
> -symbol table a concurrent hash map, so that we let multiple workers<br>
> -work on symbol table concurrently. Symbol resolution in this design is<br>
> -a single pointer mutation, which allows the resolver work concurrently<br>
> -in a lock-free manner using atomic pointer compare-and-swap.<br>
> -<br>
> -It should also be easy to apply relocations and write chunks concurrently.<br>
> -<br>
> -We created an experimental multi-threaded linker using the Microsoft<br>
> -ConcRT concurrency library, and it was able to link itself in 0.5<br>
> -seconds, so we think the design is promising.<br>
> -<br>
> -Link-Time Optimization<br>
> -----------------------<br>
> -<br>
> -LTO is implemented by handling LLVM bitcode files as object files.<br>
> -The linker resolves symbols in bitcode files normally. If all symbols<br>
> -are successfully resolved, it then calls an LLVM libLTO function<br>
> -with all bitcode files to convert them to one big regular COFF file.<br>
> -Finally, the linker replaces bitcode symbols with COFF symbols,<br>
> -so that we can link the input files as if they were in the native<br>
> -format from the beginning.<br>
> -<br>
> -The details are described in this document.<br>
> -<a href="http://llvm.org/docs/LinkTimeOptimization.html" rel="noreferrer" target="_blank">http://llvm.org/docs/LinkTimeOptimization.html</a><br>
> -<br>
> -Glossary<br>
> ---------<br>
> -<br>
> -* RVA<br>
> -<br>
> - Short for Relative Virtual Address.<br>
> -<br>
> - Windows executables or DLLs are not position-independent; they are<br>
> - linked against a fixed address called an image base. RVAs are<br>
> - offsets from an image base.<br>
> -<br>
> - Default image bases are 0x140000000 for executables and 0x18000000<br>
> - for DLLs. For example, when we are creating an executable, we assume<br>
> - that the executable will be loaded at address 0x140000000 by the<br>
> - loader, so we apply relocations accordingly. Result texts and data<br>
> - will contain raw absolute addresses.<br>
> -<br>
> -* VA<br>
> -<br>
> - Short for Virtual Address. Equivalent to RVA + image base. It is<br>
> - rarely used. We almost always use RVAs instead.<br>
> -<br>
> -* Base relocations<br>
> -<br>
> - Relocation information for the loader. If the loader decides to map<br>
> - an executable or a DLL to a different address than their image<br>
> - bases, it fixes up binaries using information contained in the base<br>
> - relocation table. A base relocation table consists of a list of<br>
> - locations containing addresses. The loader adds a difference between<br>
> - RVA and actual load address to all locations listed there.<br>
> -<br>
> - Note that this run-time relocation mechanism is much simpler than ELF.<br>
> - There's no PLT or GOT. Images are relocated as a whole just<br>
> - by shifting entire images in memory by some offsets. Although doing<br>
> - this breaks text sharing, I think this mechanism is not actually bad<br>
> - on today's computers.<br>
> -<br>
> -* ICF<br>
> -<br>
> - Short for Identical COMDAT Folding.<br>
> -<br>
> - ICF is an optimization to reduce output size by merging COMDAT sections<br>
> - by not only their names but by their contents. If two COMDAT sections<br>
> - happen to have the same metadata, actual contents and relocations,<br>
> - they are merged by ICF. It is known as an effective technique,<br>
> - and it usually reduces C++ program's size by a few percent or more.<br>
> -<br>
> - Note that this is not entirely sound optimization. C/C++ require<br>
> - different functions have different addresses. If a program depends on<br>
> - that property, it would fail at runtime. However, that's not really an<br>
> - issue on Windows because MSVC link.exe enabled the optimization by<br>
> - default. As long as your program works with the linker's default<br>
> - settings, your program should be safe with ICF.<br>
> +See docs/NewLLD.rst<br>
><br>
> Modified: lld/trunk/ELF/README.md<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/README.md?rev=263336&r1=263335&r2=263336&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/README.md?rev=263336&r1=263335&r2=263336&view=diff</a><br>
> ==============================================================================<br>
> --- lld/trunk/ELF/README.md (original)<br>
> +++ lld/trunk/ELF/README.md Sat Mar 12 00:06:40 2016<br>
> @@ -1,34 +1 @@<br>
> -The New ELF Linker<br>
> -==================<br>
> -This directory contains a port of the new PE/COFF linker for ELF.<br>
> -<br>
> -Overall Design<br>
> ---------------<br>
> -See COFF/README.md for details on the design. Note that unlike COFF, we do not<br>
> -distinguish chunks from input sections; they are merged together.<br>
> -<br>
> -Capabilities<br>
> -------------<br>
> -This linker can link LLVM and Clang on Linux/x86-64 or FreeBSD/x86-64<br>
> -"Hello world" can be linked on Linux/PPC64 and on Linux/AArch64 or<br>
> -FreeBSD/AArch64.<br>
> -<br>
> -Performance<br>
> ------------<br>
> -Achieving good performance is one of our goals. It's too early to reach a<br>
> -conclusion, but we are optimistic about that as it currently seems to be faster<br>
> -than GNU gold. It will be interesting to compare when we are close to feature<br>
> -parity.<br>
> -<br>
> -Library Use<br>
> ------------<br>
> -<br>
> -You can embed LLD to your program by linking against it and calling the linker's<br>
> -entry point function lld::elf::link.<br>
> -<br>
> -The current policy is that it is your reponsibility to give trustworthy object<br>
> -files. The function is guaranteed to return as long as you do not pass corrupted<br>
> -or malicious object files. A corrupted file could cause a fatal error or SEGV.<br>
> -That being said, you don't need to worry too much about it if you create object<br>
> -files in a usual way and give them to the linker (it is naturally expected to<br>
> -work, or otherwise it's a linker's bug.)<br>
> +See docs/NewLLD.rst<br>
><br>
> Copied: lld/trunk/docs/AtomLLD.rst (from r263292, lld/trunk/docs/index.rst)<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/docs/AtomLLD.rst?p2=lld/trunk/docs/AtomLLD.rst&p1=lld/trunk/docs/index.rst&r1=263292&r2=263336&rev=263336&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/docs/AtomLLD.rst?p2=lld/trunk/docs/AtomLLD.rst&p1=lld/trunk/docs/index.rst&r1=263292&r2=263336&rev=263336&view=diff</a><br>
> ==============================================================================<br>
> --- lld/trunk/docs/index.rst (original)<br>
> +++ lld/trunk/docs/AtomLLD.rst Sat Mar 12 00:06:40 2016<br>
> @@ -1,20 +1,14 @@<br>
> -.. _index:<br>
> +ATOM-based lld<br>
> +==============<br>
><br>
> -lld - The LLVM Linker<br>
> -=====================<br>
> -<br>
> -lld contains two linkers whose architectures are different from each other.<br>
> -One is a linker that implements native features directly.<br>
> -They are in `COFF` or `ELF` directories. Other directories contains the other<br>
> -implementation that is designed to be a set of modular code for creating<br>
> -linker tools. This document covers mainly the latter.<br>
> -For the former, please read README.md in `COFF` directory.<br>
> +ATOM-based lld is a new set of modular code for creating linker tools.<br>
> +Currently it supports Mach-O.<br>
><br>
> * End-User Features:<br>
><br>
> * Compatible with existing linker options<br>
> - * Reads standard Object Files (e.g. ELF, Mach-O, PE/COFF)<br>
> - * Writes standard Executable Files (e.g. ELF, Mach-O, PE)<br>
> + * Reads standard Object Files<br>
> + * Writes standard Executable Files<br>
> * Remove clang's reliance on "the system linker"<br>
> * Uses the LLVM `"UIUC" BSD-Style license`__.<br>
><br>
> @@ -44,29 +38,6 @@ system assembler tool, the lld project w<br>
> system linker tool.<br>
><br>
><br>
> -Current Status<br>
> ---------------<br>
> -<br>
> -lld can self host on x86-64 FreeBSD and Linux and x86 Windows.<br>
> -<br>
> -All SingleSource tests in test-suite pass on x86-64 Linux.<br>
> -<br>
> -All SingleSource and MultiSource tests in the LLVM test-suite<br>
> -pass on MIPS 32-bit little-endian Linux.<br>
> -<br>
> -Source<br>
> -------<br>
> -<br>
> -lld is available in the LLVM SVN repository::<br>
> -<br>
> - svn co <a href="http://llvm.org/svn/llvm-project/lld/trunk" rel="noreferrer" target="_blank">http://llvm.org/svn/llvm-project/lld/trunk</a> lld<br>
> -<br>
> -lld is also available via the read-only git mirror::<br>
> -<br>
> - git clone <a href="http://llvm.org/git/lld.git" rel="noreferrer" target="_blank">http://llvm.org/git/lld.git</a><br>
> -<br>
> -Put it in llvm's tools/ directory, rerun cmake, then build target lld.<br>
> -<br>
> Contents<br>
> --------<br>
><br>
><br>
> Added: lld/trunk/docs/NewLLD.rst<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/docs/NewLLD.rst?rev=263336&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/docs/NewLLD.rst?rev=263336&view=auto</a><br>
> ==============================================================================<br>
> --- lld/trunk/docs/NewLLD.rst (added)<br>
> +++ lld/trunk/docs/NewLLD.rst Sat Mar 12 00:06:40 2016<br>
> @@ -0,0 +1,309 @@<br>
> +The ELF and COFF Linkers<br>
> +========================<br>
> +<br>
> +We started rewriting the ELF (Unix) and COFF (Windows) linkers in May 2015.<br>
> +Since then, we have been making a steady progress towards providing<br>
> +drop-in replacements for the system linkers.<br>
> +<br>
> +Currently, the Windows support is mostly complete and is about 2x faster<br>
> +than the linker that comes as a part of Micrsoft Visual Studio toolchain.<br>
> +<br>
> +The ELF support is in progress and is able to link large programs<br>
> +such as Clang or LLD itself. Unless your program depends on linker scripts,<br>
> +you can expect it to be linkable with LLD.<br>
> +It is currently about 1.2x to 2x faster than GNU gold linker.<br>
> +We aim to make it a drop-in replacement for the GNU linker.<br>
> +<br>
> +We expect that FreeBSD is going to be the first large system<br>
> +to adopt LLD as the system linker.<br>
> +We are working on it in collaboration with the FreeBSD project.<br>
> +<br>
> +The linkers are notably small; as of March 2016,<br>
> +the COFF linker is under 7k LOC and the ELF linker is about 10k LOC.<br>
> +<br>
> +The linkers are designed to be as fast and simple as possible.<br>
> +Because it is simple, it is easy to extend it to support new features.<br>
> +There a few key design choices that we made to achieve these goals.<br>
> +We will describe them in this document.<br>
> +<br>
> +The ELF Linker as a Library<br>
> +---------------------------<br>
> +<br>
> +You can embed LLD to your program by linking against it and calling the linker's<br>
> +entry point function lld::elf::link.<br>
> +<br>
> +The current policy is that it is your reponsibility to give trustworthy object<br>
> +files. The function is guaranteed to return as long as you do not pass corrupted<br>
> +or malicious object files. A corrupted file could cause a fatal error or SEGV.<br>
> +That being said, you don't need to worry too much about it if you create object<br>
> +files in the usual way and give them to the linker. It is naturally expected to<br>
> +work, or otherwise it's a linker's bug.<br>
> +<br>
> +Design<br>
> +======<br>
> +<br>
> +We will describe the design of the linkers in the rest of the document.<br>
> +<br>
> +Key Concepts<br>
> +------------<br>
> +<br>
> +Linkers are fairly large pieces of software.<br>
> +There are many design choices you have to make to create a complete linker.<br>
> +<br>
> +This is a list of design choices we've made for ELF and COFF LLD.<br>
> +We believe that these high-level design choices achieved a right balance<br>
> +between speed, simplicity and extensibility.<br>
> +<br>
> +* Implement as native linkers<br>
> +<br>
> + We implemented the linkers as native linkers for each file format.<br>
> +<br>
> + The two linkers share the same design but do not share code.<br>
> + Sharing code makes sense if the benefit is worth its cost.<br>
> + In our case, ELF and COFF are different enough that we thought the layer to<br>
> + abstract the differences wouldn't worth its complexity and run-time cost.<br>
> + Elimination of the abstract layer has greatly simplified the implementation.<br>
> +<br>
> +* Speed by design<br>
> +<br>
> + One of the most important thing in archiving high performance is to<br>
> + do less rather than do it efficiently.<br>
> + Therefore, the high-level design matters more than local optimizations.<br>
> + Since we are trying to create a high-performance linker,<br>
> + it is very important to keep the design as efficient as possible.<br>
> +<br>
> + Broadly speaking, we do not do anything until we have to do it.<br>
> + For example, we do not read section contents or relocations<br>
> + until we need them to continue linking.<br>
> + When we need to do some costly operation (such as looking up<br>
> + a hash table for each symbol), we do it only once.<br>
> + We obtain a handler (which is typically just a pointer to actual data)<br>
> + on the first operation and use it throughout the process.<br>
> +<br>
> +* Efficient archive file handling<br>
> +<br>
> + LLD's handling of archive files (the files with ".a" file extension) is different<br>
> + from the traditional Unix linkers and pretty similar to Windows linkers.<br>
> + We'll describe how the traditional Unix linker handles archive files,<br>
> + what the problem is, and how LLD approached the problem.<br>
> +<br>
> + The traditional Unix linker maintains a set of undefined symbols during linking.<br>
> + The linker visits each file in the order as they appeared in the command line<br>
> + until the set becomes empty. What the linker would do depends on file type.<br>
> +<br>
> + - If the linker visits an object file, the linker links object files to the result,<br>
> + and undefined symbols in the object file are added to the set.<br>
> +<br>
> + - If the linker visits an archive file, it checks for the archive file's symbol table<br>
> + and extracts all object files that have definitions for any symbols in the set.<br>
> +<br>
> + This algorithm sometimes leads to a counter-intuitive behavior.<br>
> + If you give archive files before object files, nothing will happen<br>
> + because when the linker visits archives, there is no undefined symbols in the set.<br>
> + As a result, no files are extracted from the first archive file,<br>
> + and the link is done at that point because the set is empty after it visits one file.<br>
> +<br>
> + You can fix the problem by reordering the files,<br>
> + but that cannot fix the issue of mutually-dependent archive files.<br>
> +<br>
> + Linking mutually-dependent archive files is tricky.<br>
> + You may specify the same archive file multiple times to<br>
> + let the linker visit it more than once.<br>
> + Or, you may use the special command line options, `-(` and `-)`,<br>
> + to let the linker loop over the files between the options until<br>
> + no new symbols are added to the set.<br>
> +<br>
> + Visiting the same archive files multiple makes the linker slower.<br>
> +<br>
> + Here is how LLD approached the problem. Instead of memorizing only undefined symbols,<br>
> + we program LLD so that it memorizes all symbols.<br>
> + When it sees an undefined symbol that can be resolved by extracting an object file<br>
> + from an archive file it previously visited, it immediately extracts the file and link it.<br>
> + It is doable because LLD does not forget symbols it have seen in archive files.<br>
> +<br>
> + We believe that the LLD's way is efficient and easy to justify.<br>
> +<br>
> + The semantics of LLD's archive handling is different from the traditional Unix's.<br>
> + You can observe it if you carefully craft archive files to exploit it.<br>
> + However, in reality, we don't know any program that cannot link<br>
> + with our algorithm so far, so we are not too worried about the incompatibility.<br>
> +<br>
> +Important Data Strcutures<br>
> +-------------------------<br>
> +<br>
> +We will describe the key data structures in LLD in this section.<br>
> +The linker can be understood as the interactions between them.<br>
> +Once you understand their functions, the code of the linker should look obvious to you.<br>
> +<br>
> +* SymbolBody<br>
> +<br>
> + SymbolBody is a class to represent symbols.<br>
> + They are created for symbols in object files or archive files.<br>
> + The linker creates linker-defined symbols as well.<br>
> +<br>
> + There are basically three types of SymbolBodies: Defined, Undefined, or Lazy.<br>
> +<br>
> + - Defined symbols are for all symbols that are considered as "resolved",<br>
> + including real defined symbols, COMDAT symbols, common symbols,<br>
> + absolute symbols, linker-created symbols, etc.<br>
> + - Undefined symbols represent undefined symbols, which need to be replaced by<br>
> + Defined symbols by the resolver until the link is complete.<br>
> + - Lazy symbols represent symbols we found in archive file headers<br>
> + which can turn into Defined if we read archieve members.<br>
> +<br>
> +* Symbol<br>
> +<br>
> + Symbol is a pointer to a SymbolBody. There's only one Symbol for<br>
> + each unique symbol name (this uniqueness is guaranteed by the symbol table).<br>
> + Because SymbolBodies are created for each file independently,<br>
> + there can be many SymbolBodies for the same name.<br>
> + Thus, the relationship between Symbols and SymbolBodies is 1:N.<br>
> + You can think of Symbols as handles for SymbolBodies.<br>
> +<br>
> + The resolver keeps the Symbol's pointer to always point to the "best" SymbolBody.<br>
> + Pointer mutation is the resolve operation of this linker.<br>
> +<br>
> + SymbolBodies have pointers to their Symbols.<br>
> + That means you can always find the best SymbolBody from<br>
> + any SymbolBody by following pointers twice.<br>
> + This structure makes it very easy and cheap to find replacements for symbols.<br>
> + For example, if you have an Undefined SymbolBody, you can find a Defined<br>
> + SymbolBody for that symbol just by going to its Symbol and then to SymbolBody,<br>
> + assuming the resolver have successfully resolved all undefined symbols.<br>
> +<br>
> +* SymbolTable<br>
> +<br>
> + SymbolTable is basically a hash table from strings to Symbols<br>
> + with a logic to resolve symbol conflicts. It resolves conflicts by symbol type.<br>
> +<br>
> + - If we add Undefined and Defined symbols, the symbol table will keep the latter.<br>
> + - If we add Defined and Lazy symbols, it will keep the former.<br>
> + - If we add Lazy and Undefined, it will keep the former,<br>
> + but it will also trigger the Lazy symbol to load the archive member<br>
> + to actually resolve the symbol.<br>
> +<br>
> +* Chunk (COFF specific)<br>
> +<br>
> + Chunk represents a chunk of data that will occupy space in an output.<br>
> + Each regular section becomes a chunk.<br>
> + Chunks created for common or BSS symbols are not backed by sections.<br>
> + The linker may create chunks to append additional data to an output as well.<br>
> +<br>
> + Chunks know about their size, how to copy their data to mmap'ed outputs,<br>
> + and how to apply relocations to them.<br>
> + Specifically, section-based chunks know how to read relocation tables<br>
> + and how to apply them.<br>
> +<br>
> +* InputSection (ELF specific)<br>
> +<br>
> + Since we have less synthesized data for ELF, we don't abstract slices of<br>
> + input files as Chunks for ELF. Instead, we directly use the input section<br>
> + as an internal data type.<br>
> +<br>
> + InputSection knows about their size and how to copy themselves to<br>
> + mmap'ed outputs, just like COFF Chunks.<br>
> +<br>
> +* OutputSection<br>
> +<br>
> + OutputSection is a container of InputSections (ELF) or Chunks (COFF).<br>
> + An InputSection or Chunk belongs to at most one OutputSection.<br>
> +<br>
> +There are mainly three actors in this linker.<br>
> +<br>
> +* InputFile<br>
> +<br>
> + InputFile is a superclass of file readers.<br>
> + We have a different subclass for each input file type,<br>
> + such as regular object file, archive file, etc.<br>
> + They are responsible for creating and owning SymbolBodies and<br>
> + InputSections/Chunks.<br>
> +<br>
> +* Writer<br>
> +<br>
> + The writer is responsible for writing file headers and InputSections/Chunks to a file.<br>
> + It creates OutputSections, put all InputSections/Chunks into them,<br>
> + assign unique, non-overlapping addresses and file offsets to them,<br>
> + and then write them down to a file.<br>
> +<br>
> +* Driver<br>
> +<br>
> + The linking process is drived by the driver. The driver<br>
> +<br>
> + - processes command line options,<br>
> + - creates a symbol table,<br>
> + - creates an InputFile for each input file and put all symbols in it into the symbol table,<br>
> + - checks if there's no remaining undefined symbols,<br>
> + - creates a writer,<br>
> + - and passes the symbol table to the writer to write the result to a file.<br>
> +<br>
> +Link-Time Optimization<br>
> +----------------------<br>
> +<br>
> +LTO is implemented by handling LLVM bitcode files as object files.<br>
> +The linker resolves symbols in bitcode files normally. If all symbols<br>
> +are successfully resolved, it then calls an LLVM libLTO function<br>
> +with all bitcode files to convert them to one big regular ELF/COFF file.<br>
> +Finally, the linker replaces bitcode symbols with ELF/COFF symbols,<br>
> +so that we link the input files as if they were in the native<br>
> +format from the beginning.<br>
> +<br>
> +The details are described in this document.<br>
> +<a href="http://llvm.org/docs/LinkTimeOptimization.html" rel="noreferrer" target="_blank">http://llvm.org/docs/LinkTimeOptimization.html</a><br>
> +<br>
> +Glossary<br>
> +--------<br>
> +<br>
> +* RVA (COFF)<br>
> +<br>
> + Short for Relative Virtual Address.<br>
> +<br>
> + Windows executables or DLLs are not position-independent; they are<br>
> + linked against a fixed address called an image base. RVAs are<br>
> + offsets from an image base.<br>
> +<br>
> + Default image bases are 0x140000000 for executables and 0x18000000<br>
> + for DLLs. For example, when we are creating an executable, we assume<br>
> + that the executable will be loaded at address 0x140000000 by the<br>
> + loader, so we apply relocations accordingly. Result texts and data<br>
> + will contain raw absolute addresses.<br>
> +<br>
> +* VA<br>
> +<br>
> + Short for Virtual Address. For COFF, it is equivalent to RVA + image base.<br>
> +<br>
> +* Base relocations (COFF)<br>
> +<br>
> + Relocation information for the loader. If the loader decides to map<br>
> + an executable or a DLL to a different address than their image<br>
> + bases, it fixes up binaries using information contained in the base<br>
> + relocation table. A base relocation table consists of a list of<br>
> + locations containing addresses. The loader adds a difference between<br>
> + RVA and actual load address to all locations listed there.<br>
> +<br>
> + Note that this run-time relocation mechanism is much simpler than ELF.<br>
> + There's no PLT or GOT. Images are relocated as a whole just<br>
> + by shifting entire images in memory by some offsets. Although doing<br>
> + this breaks text sharing, I think this mechanism is not actually bad<br>
> + on today's computers.<br>
> +<br>
> +* ICF<br>
> +<br>
> + Short for Identical COMDAT Folding (COFF) or Identical Code Folding (ELF).<br>
> +<br>
> + ICF is an optimization to reduce output size by merging read-only sections<br>
> + by not only their names but by their contents. If two read-only sections<br>
> + happen to have the same metadata, actual contents and relocations,<br>
> + they are merged by ICF. It is known as an effective technique,<br>
> + and it usually reduces C++ program's size by a few percent or more.<br>
> +<br>
> + Note that this is not entirely sound optimization. C/C++ require<br>
> + different functions have different addresses. If a program depends on<br>
> + that property, it would fail at runtime.<br>
> +<br>
> + On Windows, that's not really an issue because MSVC link.exe enabled<br>
> + the optimization by default. As long as your program works<br>
> + with the linker's default settings, your program should be safe with ICF.<br>
> +<br>
> + On Unix, your program is generally not guaranteed to be safe with ICF,<br>
> + although large programs happen to work correctly.<br>
> + LLD works fine with ICF for example.<br>
><br>
> Modified: lld/trunk/docs/index.rst<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/docs/index.rst?rev=263336&r1=263335&r2=263336&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/docs/index.rst?rev=263336&r1=263335&r2=263336&view=diff</a><br>
> ==============================================================================<br>
> --- lld/trunk/docs/index.rst (original)<br>
> +++ lld/trunk/docs/index.rst Sat Mar 12 00:06:40 2016<br>
> @@ -4,55 +4,12 @@ lld - The LLVM Linker<br>
> =====================<br>
><br>
> lld contains two linkers whose architectures are different from each other.<br>
> -One is a linker that implements native features directly.<br>
> -They are in `COFF` or `ELF` directories. Other directories contains the other<br>
> -implementation that is designed to be a set of modular code for creating<br>
> -linker tools. This document covers mainly the latter.<br>
> -For the former, please read README.md in `COFF` directory.<br>
><br>
> -* End-User Features:<br>
> -<br>
> - * Compatible with existing linker options<br>
> - * Reads standard Object Files (e.g. ELF, Mach-O, PE/COFF)<br>
> - * Writes standard Executable Files (e.g. ELF, Mach-O, PE)<br>
> - * Remove clang's reliance on "the system linker"<br>
> - * Uses the LLVM `"UIUC" BSD-Style license`__.<br>
> -<br>
> -* Applications:<br>
> -<br>
> - * Modular design<br>
> - * Support cross linking<br>
> - * Easy to add new CPU support<br>
> - * Can be built as static tool or library<br>
> -<br>
> -* Design and Implementation:<br>
> -<br>
> - * Extensive unit tests<br>
> - * Internal linker model can be dumped/read to textual format<br>
> - * Additional linking features can be plugged in as "passes"<br>
> - * OS specific and CPU specific code factored out<br>
> -<br>
> -Why a new linker?<br>
> ------------------<br>
> -<br>
> -The fact that clang relies on whatever linker tool you happen to have installed<br>
> -means that clang has been very conservative adopting features which require a<br>
> -recent linker.<br>
> -<br>
> -In the same way that the MC layer of LLVM has removed clang's reliance on the<br>
> -system assembler tool, the lld project will remove clang's reliance on the<br>
> -system linker tool.<br>
> -<br>
> -<br>
> -Current Status<br>
> ---------------<br>
> -<br>
> -lld can self host on x86-64 FreeBSD and Linux and x86 Windows.<br>
> -<br>
> -All SingleSource tests in test-suite pass on x86-64 Linux.<br>
> +.. toctree::<br>
> + :maxdepth: 1<br>
><br>
> -All SingleSource and MultiSource tests in the LLVM test-suite<br>
> -pass on MIPS 32-bit little-endian Linux.<br>
> + NewLLD<br>
> + AtomLLD<br>
><br>
> Source<br>
> ------<br>
> @@ -66,25 +23,3 @@ lld is also available via the read-only<br>
> git clone <a href="http://llvm.org/git/lld.git" rel="noreferrer" target="_blank">http://llvm.org/git/lld.git</a><br>
><br>
> Put it in llvm's tools/ directory, rerun cmake, then build target lld.<br>
> -<br>
> -Contents<br>
> ---------<br>
> -<br>
> -.. toctree::<br>
> - :maxdepth: 2<br>
> -<br>
> - design<br>
> - getting_started<br>
> - ReleaseNotes<br>
> - development<br>
> - windows_support<br>
> - open_projects<br>
> - sphinx_intro<br>
> -<br>
> -Indices and tables<br>
> -------------------<br>
> -<br>
> -* :ref:`genindex`<br>
> -* :ref:`search`<br>
> -<br>
> -__ <a href="http://llvm.org/docs/DeveloperPolicy.html#license" rel="noreferrer" target="_blank">http://llvm.org/docs/DeveloperPolicy.html#license</a><br>
><br>
><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</div></div></blockquote></div><br></div>