<div><font face="arial, sans-serif">I had some really major concerns about the general organization and wording of the beginning. I've attached a patch that reworks it to be a lot more upfront about the organization of the components and exactly what is involved. Let me know what you think.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I think I may have gone a bit overboard with the sphinx markup. That can easily be dialed back to be more "plaintext"-y (although you should build it with sphinx to see how it turns out, and decide whether it may be desirable to use some of Sphinx's features). Needless to say, I don't expect the patch to be committed as-is.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">For the middle--end, the patch has some little formatting fixups that I always compulsively do as I read. You can ignore them, but I think there might be a couple grammatical or wording fixes worth applying.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I think the Making Atoms discussion needs to be made more concrete. "Call this function". "Use this constructor", etc.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Similarly for the testing stuff. "Put a file here". "make this build target to run the tests". etc.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Also, you should link Readers.rst into a toctree; sphinx is warning about it not being linked in (you do link to it but not through the toctree; I would just remove the other link).</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">--Sean Silva</font></div><div><br></div>Having moved this information higher up, you should leave this out.<br><div class="gmail_quote">
On Tue, Jun 12, 2012 at 3:43 PM, Nick Kledzik <span dir="ltr"><<a href="mailto:kledzik@apple.com" target="_blank">kledzik@apple.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Author: kledzik<br>
Date: Tue Jun 12 17:43:35 2012<br>
New Revision: 158374<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=158374&view=rev" target="_blank">http://llvm.org/viewvc/llvm-project?rev=158374&view=rev</a><br>
Log:<br>
Wrote initial doc on how to create a Reader<br>
<br>
Added:<br>
lld/trunk/docs/Readers.rst<br>
Modified:<br>
lld/trunk/docs/development.rst<br>
<br>
Added: lld/trunk/docs/Readers.rst<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/docs/Readers.rst?rev=158374&view=auto" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/docs/Readers.rst?rev=158374&view=auto</a><br>
==============================================================================<br>
--- lld/trunk/docs/Readers.rst (added)<br>
+++ lld/trunk/docs/Readers.rst Tue Jun 12 17:43:35 2012<br>
@@ -0,0 +1,163 @@<br>
+.. _Readers:<br>
+<br>
+Developing lld Readers<br>
+======================<br>
+<br>
+Introduction<br>
+------------<br>
+<br>
+One goal of lld is to be file format independent. This is done<br>
+through a plug-in model for reading object files. The lld::Reader is the base<br>
+class for all object file readers. A Reader follows the factory method pattern.<br>
+A Reader instantiates an lld::File object (which is a graph of Atoms) from a<br>
+given object file (on disk or in-memory).<br>
+<br>
+Every Reader subclass defines its own "options" class (for instance the mach-o<br>
+Reader defines the class ReaderOptionsMachO). This options class is the<br>
+one-and-only way to control how the Reader operates when parsing an input file<br>
+into an Atom graph. For instance, you may want the Reader to only accept<br>
+certain architectures. The options class can be instantiated from command<br>
+line options, or it can be subclassed and the ivars programmatically set.<br>
+<br>
+<br>
+Where to start<br>
+--------------<br>
+<br>
+The lld project already has a skeleton of source code for Readers of ELF, COFF,<br>
+mach-o, and the lld native object file format. If your file format is a<br>
+variant of one of those, you should modify the existing Reader to support<br>
+your variant. This is done by adding new ivar(s) to the Options class for that<br>
+Reader which specifies which file format variant to expect. And then modifying<br>
+the Reader to check those ivars and respond parse the object file accordingly.<br>
+<br>
+If your object file format is not a variant of any existing Reader, you'll need<br>
+to create a new Reader subclass. If your file format is called "Foo", you'll<br>
+need to create these files::<br>
+<br>
+ ./include/lld/ReaderWriter/ReaderFoo.h<br>
+ ./lib/ReaderWriter/Foo/ReaderFoo.cpp<br>
+<br>
+The public interface for you reader is just the ReaderOptions subclass<br>
+(e.g. ReaderOptionsFoo) and the function to create a Reader given the options::<br>
+<br>
+ Reader* createReaderFoo(const ReaderOptionsFoo &options);<br>
+<br>
+In the implementation, you can define a ReaderFoo class, but that class is<br>
+private to your ReaderWriter directory.<br>
+<br>
+<br>
+Readers are factories<br>
+---------------------<br>
+<br>
+The linker will usually only instantiate your Reader once. That one Reader will<br>
+have its parseFile() method called many times with different input files.<br>
+To support a multithreaded linking, the Reader may be parsing multiple input<br>
+files in parallel. Therefore, there should be no parsing state in you Reader<br>
+object. Any parsing state should be in ivars of your File subclass or in<br>
+some temporary object.<br>
+<br>
+The key method to implement in a reader is::<br>
+<br>
+ virtual error_code parseFile(std::unique_ptr<MemoryBuffer> mb,<br>
+ std::vector<std::unique_ptr<File>> &result);<br>
+<br>
+It takes a memory buffer (which contains the contents of the object file<br>
+being read) and returns an instantiated lld::File object which is<br>
+a collection of Atoms. The result is a vector of File pointers (instead of<br>
+simple a File pointer) because some file formats allow multiple object<br>
+"files" to be encoded in one file system file.<br>
+<br>
+<br>
+Memory Ownership<br>
+----------------<br>
+<br>
+If parseFile() is successful, it either passes ownership of the MemoryBuffer<br>
+to the File object, or it deletes the MemoryBuffer. The former is done if the<br>
+Atoms contain pointers into the MemoryBuffer (e.g. StringRefs for symbols<br>
+or ArrayRefs for section content). If parseFile() fails, the MemoryBuffer<br>
+must be deleted by the Reader.<br>
+<br>
+Atoms objects are always owned by their File object. During core linking<br>
+when Atoms are coalesced or dead stripped away, core linking does not delete<br>
+those Atoms. Core linking just removes those unused Atoms from its internal<br>
+list. The destructor of a File object is responsible for deleting all Atoms<br>
+it owns, and if ownership of the MemoryBuffer was passed to it, the File<br>
+destructor needs to delete that too.<br>
+<br>
+<br>
+Making Atoms<br>
+------------<br>
+<br>
+The internal model of lld is purely Atom based. But most object files do not<br>
+have an explicit concept of Atoms, instead most have "sections". The way<br>
+to think of this, is that a section is just list of Atoms with common<br>
+attributes.<br>
+<br>
+The first step in parsing section based object files is to cleave each<br>
+section into a list of Atoms. The technique may vary by section type. For<br>
+code sections (e.g. .text), there are usually symbols at the start of each<br>
+function. Those symbol address are the points at which the section is cleaved<br>
+into discrete Atoms. Some file formats (like ELF) also include the<br>
+length of each symbol in the symbol table. Otherwise, the length of each<br>
+Atom is calculated to run to the start of the next symbol or the end of the<br>
+section.<br>
+<br>
+Other sections types can be implicitly cleaved. For instance c-string literals<br>
+or unwind info (e.g. .eh_frame) can be cleaved by having the Reader look at<br>
+the content of the section. It is important to cleave sections into Atoms<br>
+to remove false dependencies. For instance the .eh_frame section often<br>
+has no symbols, but contains "pointers" to the functions for which it<br>
+has unwind info. If the .eh_frame section was not cleaved (but left as one<br>
+big Atom), there would always be a reference (from the eh_frame Atom) to<br>
+each function. So the linker would be unable to coalesce or dead stripped<br>
+away the function atoms.<br>
+<br>
+The lld Atom model also requires that a reference to an undefined symbol be<br>
+modeled as a Reference to an UndefinedAtom. So the Reader also needs to<br>
+create an UndefinedAtom for each undefined symbol in the object file.<br>
+<br>
+Once all Atoms have been created, the second step is to create References<br>
+(recall that Atoms are "nodes" and References are "edges"). Most References<br>
+are created by looking at the "relocation records" in the object file. If<br>
+a function contains a call to "malloc", there is usually a relocation record<br>
+specifying the address in the section and the symbol table index. Your<br>
+Reader will need to convert the address to an Atom and offset and the symbol<br>
+table index into a target Atom. If "malloc" is not defined in the object file,<br>
+the target Atom of the Reference will be an UndefinedAtom.<br>
+<br>
+<br>
+Performance<br>
+-----------<br>
+Once you have the above working to parse an object file into Atoms and<br>
+References, you'll want to look at performance. Some techniques that can<br>
+help performance are:<br>
+<br>
+* Use llvm::BumpPtrAllocator or pre-allocate one big vector<Reference> and then<br>
+ just have each atom point to its subrange of References in that vector.<br>
+ This can be faster that allocating each Reference as separate object.<br>
+* Pre-scan the symbol table and determine how many atoms are in each section<br>
+ then allocate space for all the Atom objects at once.<br>
+* Don't copy symbol names or section content to each Atom, instead use<br>
+ StringRef and ArrayRef in each Atom to point to its name and content in the<br>
+ MemoryBuffer.<br>
+<br>
+<br>
+Testing<br>
+-------<br>
+<br>
+We are still working on infrastructure to test Readers. The issue is that<br>
+you don't want to check in binary files to the test suite. And the tools<br>
+for creating your object file from assembly source may not be available on<br>
+every OS.<br>
+<br>
+We are investigating a way to use yaml to describe the section, symbols,<br>
+and content of a file. Then have some code which will write out an object<br>
+file from that yaml description.<br>
+<br>
+Once that is in place, you can write test cases that contain section/symbols<br>
+yaml and is run through the linker to produce Atom/References based yaml which<br>
+is then run through FileCheck to verify the Atoms and References are as<br>
+expected.<br>
+<br>
+<br>
+<br>
<br>
Modified: lld/trunk/docs/development.rst<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/docs/development.rst?rev=158374&r1=158373&r2=158374&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/docs/development.rst?rev=158374&r1=158373&r2=158374&view=diff</a><br>
==============================================================================<br>
--- lld/trunk/docs/development.rst (original)<br>
+++ lld/trunk/docs/development.rst Tue Jun 12 17:43:35 2012<br>
@@ -5,7 +5,15 @@<br>
<br>
lld is developed as part of the `LLVM <<a href="http://llvm.org" target="_blank">http://llvm.org</a>>`_ project.<br>
<br>
-See the :ref:`getting started <getting_started>` guide.<br>
+Creating a Reader<br>
+-----------------<br>
+<br>
+See the :ref:`Creating a Reader <Readers>` guide.<br>
+<br>
+<br>
+<br>
+Documentation<br>
+-------------<br>
<br>
The project documentation is written in reStructuredText and generated using the<br>
`Sphinx <<a href="http://sphinx.pocoo.org/" target="_blank">http://sphinx.pocoo.org/</a>>`_ documentation generator. For more<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br>