<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Oct 30, 2014, at 5:09 PM, Shankar Easwaran <<a href="mailto:shankare@codeaurora.org">shankare@codeaurora.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
At the LLVM developers conference, We (Myself/Bigcheese) were
discussing one of the ways to cleanup the InputGraph and the
different kinds of filenodes that lld has internally.<br>
<br>
lld::FileLocator<br>
-----------------<br>
Motivation: In the current design, the resolver deals with files
that are already in parsed state, but there is a need that several
files may need to be read after a native object file is
read(COFF/ELF on PS4).<br>
<br>
lld would have a FileLocator with the below interface :-<br>
<br>
class FileLocator {<br>
<br>
enum Kind {<br>
InputPath,<br>
LibraryPath // The user has specified the path using -l
switches.<br>
Memory // File is in memory<br>
}<br>
<br>
// Kind = MemoryBuffer<br>
explicit FileLocator(std::unique_ptr<MemoryBuffer>);<br>
<br>
// Kind = InputPath<br>
explicit FileLocator(StringRef path);<br>
<br>
<br>
private:<br>
std::unique_ptr<MemoryBuffer> _buffer;<br>
StringRef _path;<br>
};<br>
<br>
Simplify InputGraph<br>
--------------------<br>
Motivation: Discussion on InputGraph in the lld mailing list, that
the current InputGraph interface is complicated.<br>
<br>
Tags<br>
^^^^^<br>
The InputGraph can only be simplified when the Resolver would be
able to handle flavor specific tags. All the positional parameters
the driver would handle would have a tag associated with the
FileLocator. Examples of tags are which affects the Resolver (or)
which affects the way an input file is handled.<br>
<br>
Examples of tags for the Gnu Flavor are below :-<br>
<br>
enum class Tag {<br>
StartGroup,<br>
EndGroup,<br>
PreferSharedLibrary,<br>
PrefererStaticLibrary<br>
};<br></div></div></blockquote><div>I don’t understand these tags. Seems like the [Start|End]Group are *between* FileLocators (just like on the command line). What is the Prefer* tag? Does it apply just to that FileLocator, or all follow on ones?</div><div><br></div><div>What about other attributes like whole-archive?</div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">
<br>
InputElement<br>
^^^^^^^^^^^^^^<br>
The InputElement class will be simplified which would contain a
tag and the FileLocator.<br>
<br>
struct InputElement {<br>
Tag _tag;<br>
FileLocator _locator;<br>
};<br>
<br>
LinkingContext<br>
^^^^^^^^^^^^^^^^<br>
LinkingContext would have a vector of InputElements, which the
driver would process and parse.<br></div></div></blockquote><div>I do like the goal of a one dimensional list instead of a graph. But hard part is a design that works for all flavors...</div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">
<br>
The Driver would create InputElements and add to the vector of
InputElements in the LinkingContext.<br>
<br>
FileManager<br>
-------------<br>
All files would be accessed using a FileManager interface that is
an interface with the FileLocator and handles conversion of a
Input file to a MemoryBuffer.<br>
<br>
A new class that interfaces with the Driver will need to be
created for searching input files depending on the user search
path provided to lld.<br>
<br>
class DriverInterface {<br>
std::string getAbsolutePath(FileLocator &loc) const = 0;<br>
};<br></div></div></blockquote><div>Why can’t the driver pre-resolve all paths so that FileLocator just deals with full paths and not searching? </div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">
<br>
class FileManager {<br>
explicit FileManager(DriverInterface &d);<br>
MemoryBuffer &getBuffer(FileLocator &loc);<br>
const DriverInterface& _interface;<br>
};<br>
<br></div></div></blockquote><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">Resolver<br>
---------<br>
The Resolver processes InputElements and the tags relevant for the
resolver.<br>
<br>
Currently start/end group tags are the only ones I think are
needed to be supported by the resolver.<br></div></div></blockquote><div>I don’t see how mach-o linking will work. The driver needs to take many passes over the whole list. The first pass just picks out object files. The remaining passes only pick out libraries and it repeats until nothing new is loaded.</div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">
<br>
Since the resolver processes InputElements which may not be
parsed, the Resolver would access the registry to convert the
InputElement to a parsed File representation.<br></div></div></blockquote><div>When does parsing happen? When would it not happen (as it does today) before the Resolver is even run?</div><div><br></div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix">
<br>
lld::File<br>
----------<br>
class File {<br>
range<InputElement> getInputs();<br>
}<br>
<br>
Every lld::File will return a range of InputElements that may need
to be additionally parsed when a file is read.<br></div></div></blockquote><div>This is where it gets interesting. The ability to have multiple “files” in one yaml disk file is great for writing one-file-test-cases. But now every disk file may contain multiple logical lld::Files some of which may be relocatable or library files. The Resolver just needs an iterator over the lld::Files, but the driver is providing the disk files.</div><div><br></div><div><br></div><div>I still kinda of like my idea of the LinkingContext having a virtual forEachFile() method that takes a lambda. The lambda is called with the lld::File to process and it returns a bool if anything from the file was used. The implementation of forEachFile() for each flavor knows how to walk the input files list in the right way for its semantics. The ELFLinkingContext would know where the groups are and repeat the files in the group until the lambda returns no files were used. The MachOLinkingContext would know to make one pass on all input files calling the lambda on only relocatable files, then only call it on library files.</div><div><br></div><div>-Nick</div><div><br></div><div><br></div><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><div class="moz-cite-prefix"><br>
<br>
On 10/22/2014 8:16 PM, Shankar Easwaran wrote:<br>
</div>
<blockquote cite="mid:544856E9.6040900@codeaurora.org" type="cite">On
10/22/2014 8:01 PM, Rui Ueyama wrote:
<br>
<blockquote type="cite">On Tue Oct 21 2014 at 9:38:57 PM Nick
Kledzik <a class="moz-txt-link-rfc2396E" href="mailto:kledzik@apple.com"><kledzik@apple.com></a> wrote:
<br>
<br>
<blockquote type="cite">On Oct 21, 2014, at 6:31 PM, Rui Ueyama
<a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<br>
Let's not add a back reference to the input element. It will
make the
<br>
complicated data structure even worse and would raise new
issues (for
<br>
example, I could imagine that it's not clear whether the back
reference
<br>
should point to a parsed file in an archive file or should
point to the
<br>
entire archive file.)
<br>
<br>
Before discussing possible alternatives, it's worth discussing
the issues
<br>
that the current InputGraph has. Here's my take.
<br>
<br>
1. We have too many classes to represent the concept of
"file".
<br>
<br>
FileNode in the InputGraph is different from the File class
that
<br>
represents parsed file. And the InputGraph has various kinds
of FileNodes
<br>
-- FileNode, SimpleFileNode, <flavor>FileNode, etc. That
needless
<br>
distinction makes hard to handle files.
<br>
<br>
FileNode needs to be merged with the File class. Or if we are
going to
<br>
replace the InputGraph with a simpler data structure, it
should be removed.
<br>
<br>
2. We have too many (other) classes.
<br>
<br>
We have too many classes for the InputGraph that are simply
overdesigned.
<br>
Command line argument is not an easy stuff, but it shouldn't
need that
<br>
amount of code and number of classes. We have created too deep
class
<br>
hierarchy there.
<br>
<br>
3. Making the core linker platform neutral is not an ultimate
goal, but
<br>
the InputGraph is designed for that.
<br>
<br>
The InputGraph made made the core linker more or less platform
neutral.
<br>
That's basically a good thing. We'd like to keep the core part
as platform
<br>
neutral as possible because it would help us maintain the
code. However,
<br>
neutrality is not a goal that we need to achieve at any cost.
If separating
<br>
platform neutral code from dependent code will mess up the
entire code
<br>
base, we simply shouldn't do that. (It's worth noting that the
InputGraph
<br>
failed to hide everything already, one example is the ELF
section group.)
<br>
<br>
We should teach the core linker a little bit more about
platform specific
<br>
features and removes complexity from the InputGraph.
<br>
<br>
I agree on all the above.
<br>
<br>
Looking at how core linking uses InputGraph, it is always
through the
<br>
LinkingContext and:
<br>
Resolver.cpp needs an iterator (getNextFile()) and a way to
notify when
<br>
it used a file (notifyProgress())
<br>
Driver.cpp needs a way to iterate files (inputElements) to
parse them in
<br>
parallel
<br>
<br>
So this could be simplified to hide/implement the “graph” in
the
<br>
LinkingContext. That is, the LinkingContext subclasses each
use the data
<br>
structures natural for their needs. The drivers just call
addFile() on the
<br>
LinkingContext, and the gnu driver calls
startGroup()/endGroup() on the
<br>
LinkingContext. The Resolver could call a forEachFile()
method on
<br>
LinkingContext which takes a lambda on what to do with each
file. The
<br>
lambda can return if the Resolver used the file (to track
whether the
<br>
current group should be scanned again). The parallel parsing
currently done
<br>
in the Driver could be a utility method in the base
LinkingContext class
<br>
that each concrete LinkingContext uses.
<br>
<br>
</blockquote>
Why does the GNU ld driver have to ask the LinkingContext to
construct a
<br>
startGroup/endGroup? The GNU driver should be capable of
understanding all
<br>
the command line options for GNU ld, so it can construct a data
structure
<br>
directly.
<br>
<br>
forEachFile() doesn't seem different from what we have now. It's
a
<br>
getNextFile(). I don't think it will improve the situation.
<br>
<br>
It feels to me that we don't need a sophisticated technique
here. I'd
<br>
suggest making the Resolver to handle ELF groups to eliminate
<br>
notifyProgress. I'd even remove InputGraph entirely and replace
it with a
<br>
std::vector<FileOrGroup> or something like that where
FileOrGroup is a
<br>
class representing a file or a ELF group.
<br>
</blockquote>
I dont think the Core resolver should not understand anything
about ELF.
<br>
<br>
Even changing this to a std::vector<FileOrGroup> doesnot
change what can be achieved with the new functionality compared to
what we have now.
<br>
<br>
As this is one of the major things that we need to handle, I think
We could move this discusssion to a hallway discussion during the
LLVM developers meeting, if you are all ok.
<br>
<blockquote type="cite">
<br>
<blockquote type="cite">
<br>
On Tue, Oct 21, 2014 at 5:33 PM, Shankar Easwaram
<a class="moz-txt-link-rfc2396E" href="mailto:shankarke@gmail.com"><shankarke@gmail.com></a>
<br>
wrote:
<br>
<br>
<blockquote type="cite">If we could have lld::file have a back
reference to the input element
<br>
(or) use the file ordinal to access the input element in the
input graph
<br>
you could just add inputs on a need basis ?
<br>
<br>
Let's discuss alternative proposals as part of this thread ?
<br>
<br>
I am also interested in moving the input file which could be
a linker
<br>
script from the driver to the registry too in addition.
<br>
<br>
We still need to have the getNextFile as it made the
resolver more flavor
<br>
neutral.
<br>
<br>
Shankar easwaran
<br>
<br>
<br>
<br>
On Oct 21, 2014, at 17:58, Rui Ueyama
<a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<br>
On Tue, Oct 21, 2014 at 3:41 PM, Michael Spencer
<a class="moz-txt-link-rfc2396E" href="mailto:bigcheesegs@gmail.com"><bigcheesegs@gmail.com></a>
<br>
wrote:
<br>
<br>
<blockquote type="cite">On Mon, Oct 20, 2014 at 9:21 PM, Rui
Ueyama <a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<blockquote type="cite">I'm not still sure whether or not
"InputGraph" thing was after all the
<br>
</blockquote>
right way to abstract the input file list. That often
makes easy things
<br>
hard. In this case, sorting a list of input files became
that hard. When I
<br>
proposed the idea I was thinking that would make this kind
of things easier
<br>
to handle. I don't think the current shape of the API is
not desirable at
<br>
least. Or the fundamental idea was not very good.
<br>
<blockquote type="cite">One thing we probably should
consider is, in my opinion, we are
<br>
</blockquote>
pushing too hard to separate all ports. We are trying to
write any
<br>
architecture-dependent code into architecture-specific
file. But the
<br>
natural border of API doesn't always fit to the
architecture
<br>
dependent/independent border. We should probably relax
that constraint a
<br>
bit where that makes sense and write code that's short and
easier to
<br>
understand.
<br>
<br>
I agree. The current InputGraph code makes modifying the
input very
<br>
difficult to do. Part of the reason for the model in the
first place
<br>
was to make this easy. I'm currently working on adding
#pragma lib
<br>
like support for ELF (PS4 specific stuff) and have found
it basically
<br>
impossible to add a new input file in the correct position
in the
<br>
graph with the current model.
<br>
<br>
</blockquote>
To deal with the issue of the InputGraph I have a few ugly
hacks in
<br>
PECOFF/LinkerGeneratedSymbolFile.h. There are a few virtual
.a files there
<br>
that generate magical symbols on the fly to control the core
linker in an
<br>
obscure way. These hacks need to be removed, but because the
InputFile API
<br>
exists in between, it's impossible to do.
<br>
<br>
The InputGraph API design needs to be revisited and probably
be rewritten.
<br>
<br>
<br>
</blockquote>
</blockquote>
<br>
<br>
_______________________________________________
<br>
llvm-commits mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a>
<br>
</blockquote>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
llvm-commits mailing list
<a class="moz-txt-link-abbreviated" href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>
<a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation</pre>
</div>
</blockquote></div><br></body></html>