<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
At the LLVM developers conference, We (Myself/Bigcheese) were
discussing one of the ways to cleanup the InputGraph and the
different kinds of filenodes that lld has internally.<br>
<br>
lld::FileLocator<br>
-----------------<br>
Motivation: In the current design, the resolver deals with files
that are already in parsed state, but there is a need that several
files may need to be read after a native object file is
read(COFF/ELF on PS4).<br>
<br>
lld would have a FileLocator with the below interface :-<br>
<br>
class FileLocator {<br>
<br>
enum Kind {<br>
InputPath,<br>
LibraryPath // The user has specified the path using -l
switches.<br>
Memory // File is in memory<br>
}<br>
<br>
// Kind = MemoryBuffer<br>
explicit FileLocator(std::unique_ptr<MemoryBuffer>);<br>
<br>
// Kind = InputPath<br>
explicit FileLocator(StringRef path);<br>
<br>
<br>
private:<br>
std::unique_ptr<MemoryBuffer> _buffer;<br>
StringRef _path;<br>
};<br>
<br>
Simplify InputGraph<br>
--------------------<br>
Motivation: Discussion on InputGraph in the lld mailing list, that
the current InputGraph interface is complicated.<br>
<br>
Tags<br>
^^^^^<br>
The InputGraph can only be simplified when the Resolver would be
able to handle flavor specific tags. All the positional parameters
the driver would handle would have a tag associated with the
FileLocator. Examples of tags are which affects the Resolver (or)
which affects the way an input file is handled.<br>
<br>
Examples of tags for the Gnu Flavor are below :-<br>
<br>
enum class Tag {<br>
StartGroup,<br>
EndGroup,<br>
PreferSharedLibrary,<br>
PrefererStaticLibrary<br>
};<br>
<br>
InputElement<br>
^^^^^^^^^^^^^^<br>
The InputElement class will be simplified which would contain a
tag and the FileLocator.<br>
<br>
struct InputElement {<br>
Tag _tag;<br>
FileLocator _locator;<br>
};<br>
<br>
LinkingContext<br>
^^^^^^^^^^^^^^^^<br>
LinkingContext would have a vector of InputElements, which the
driver would process and parse.<br>
<br>
The Driver would create InputElements and add to the vector of
InputElements in the LinkingContext.<br>
<br>
FileManager<br>
-------------<br>
All files would be accessed using a FileManager interface that is
an interface with the FileLocator and handles conversion of a
Input file to a MemoryBuffer.<br>
<br>
A new class that interfaces with the Driver will need to be
created for searching input files depending on the user search
path provided to lld.<br>
<br>
class DriverInterface {<br>
std::string getAbsolutePath(FileLocator &loc) const = 0;<br>
};<br>
<br>
class FileManager {<br>
explicit FileManager(DriverInterface &d);<br>
MemoryBuffer &getBuffer(FileLocator &loc);<br>
const DriverInterface& _interface;<br>
};<br>
<br>
Resolver<br>
---------<br>
The Resolver processes InputElements and the tags relevant for the
resolver.<br>
<br>
Currently start/end group tags are the only ones I think are
needed to be supported by the resolver.<br>
<br>
Since the resolver processes InputElements which may not be
parsed, the Resolver would access the registry to convert the
InputElement to a parsed File representation.<br>
<br>
lld::File<br>
----------<br>
class File {<br>
range<InputElement> getInputs();<br>
}<br>
<br>
Every lld::File will return a range of InputElements that may need
to be additionally parsed when a file is read.<br>
<br>
Please raise your opinions and thoughts on the design.<br>
<br>
Shankar Easwaran<br>
<br>
<br>
<br>
On 10/22/2014 8:16 PM, Shankar Easwaran wrote:<br>
</div>
<blockquote cite="mid:544856E9.6040900@codeaurora.org" type="cite">On
10/22/2014 8:01 PM, Rui Ueyama wrote:
<br>
<blockquote type="cite">On Tue Oct 21 2014 at 9:38:57 PM Nick
Kledzik <a class="moz-txt-link-rfc2396E" href="mailto:kledzik@apple.com"><kledzik@apple.com></a> wrote:
<br>
<br>
<blockquote type="cite">On Oct 21, 2014, at 6:31 PM, Rui Ueyama
<a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<br>
Let's not add a back reference to the input element. It will
make the
<br>
complicated data structure even worse and would raise new
issues (for
<br>
example, I could imagine that it's not clear whether the back
reference
<br>
should point to a parsed file in an archive file or should
point to the
<br>
entire archive file.)
<br>
<br>
Before discussing possible alternatives, it's worth discussing
the issues
<br>
that the current InputGraph has. Here's my take.
<br>
<br>
1. We have too many classes to represent the concept of
"file".
<br>
<br>
FileNode in the InputGraph is different from the File class
that
<br>
represents parsed file. And the InputGraph has various kinds
of FileNodes
<br>
-- FileNode, SimpleFileNode, <flavor>FileNode, etc. That
needless
<br>
distinction makes hard to handle files.
<br>
<br>
FileNode needs to be merged with the File class. Or if we are
going to
<br>
replace the InputGraph with a simpler data structure, it
should be removed.
<br>
<br>
2. We have too many (other) classes.
<br>
<br>
We have too many classes for the InputGraph that are simply
overdesigned.
<br>
Command line argument is not an easy stuff, but it shouldn't
need that
<br>
amount of code and number of classes. We have created too deep
class
<br>
hierarchy there.
<br>
<br>
3. Making the core linker platform neutral is not an ultimate
goal, but
<br>
the InputGraph is designed for that.
<br>
<br>
The InputGraph made made the core linker more or less platform
neutral.
<br>
That's basically a good thing. We'd like to keep the core part
as platform
<br>
neutral as possible because it would help us maintain the
code. However,
<br>
neutrality is not a goal that we need to achieve at any cost.
If separating
<br>
platform neutral code from dependent code will mess up the
entire code
<br>
base, we simply shouldn't do that. (It's worth noting that the
InputGraph
<br>
failed to hide everything already, one example is the ELF
section group.)
<br>
<br>
We should teach the core linker a little bit more about
platform specific
<br>
features and removes complexity from the InputGraph.
<br>
<br>
I agree on all the above.
<br>
<br>
Looking at how core linking uses InputGraph, it is always
through the
<br>
LinkingContext and:
<br>
Resolver.cpp needs an iterator (getNextFile()) and a way to
notify when
<br>
it used a file (notifyProgress())
<br>
Driver.cpp needs a way to iterate files (inputElements) to
parse them in
<br>
parallel
<br>
<br>
So this could be simplified to hide/implement the “graph” in
the
<br>
LinkingContext. That is, the LinkingContext subclasses each
use the data
<br>
structures natural for their needs. The drivers just call
addFile() on the
<br>
LinkingContext, and the gnu driver calls
startGroup()/endGroup() on the
<br>
LinkingContext. The Resolver could call a forEachFile()
method on
<br>
LinkingContext which takes a lambda on what to do with each
file. The
<br>
lambda can return if the Resolver used the file (to track
whether the
<br>
current group should be scanned again). The parallel parsing
currently done
<br>
in the Driver could be a utility method in the base
LinkingContext class
<br>
that each concrete LinkingContext uses.
<br>
<br>
</blockquote>
Why does the GNU ld driver have to ask the LinkingContext to
construct a
<br>
startGroup/endGroup? The GNU driver should be capable of
understanding all
<br>
the command line options for GNU ld, so it can construct a data
structure
<br>
directly.
<br>
<br>
forEachFile() doesn't seem different from what we have now. It's
a
<br>
getNextFile(). I don't think it will improve the situation.
<br>
<br>
It feels to me that we don't need a sophisticated technique
here. I'd
<br>
suggest making the Resolver to handle ELF groups to eliminate
<br>
notifyProgress. I'd even remove InputGraph entirely and replace
it with a
<br>
std::vector<FileOrGroup> or something like that where
FileOrGroup is a
<br>
class representing a file or a ELF group.
<br>
</blockquote>
I dont think the Core resolver should not understand anything
about ELF.
<br>
<br>
Even changing this to a std::vector<FileOrGroup> doesnot
change what can be achieved with the new functionality compared to
what we have now.
<br>
<br>
As this is one of the major things that we need to handle, I think
We could move this discusssion to a hallway discussion during the
LLVM developers meeting, if you are all ok.
<br>
<blockquote type="cite">
<br>
<blockquote type="cite">
<br>
On Tue, Oct 21, 2014 at 5:33 PM, Shankar Easwaram
<a class="moz-txt-link-rfc2396E" href="mailto:shankarke@gmail.com"><shankarke@gmail.com></a>
<br>
wrote:
<br>
<br>
<blockquote type="cite">If we could have lld::file have a back
reference to the input element
<br>
(or) use the file ordinal to access the input element in the
input graph
<br>
you could just add inputs on a need basis ?
<br>
<br>
Let's discuss alternative proposals as part of this thread ?
<br>
<br>
I am also interested in moving the input file which could be
a linker
<br>
script from the driver to the registry too in addition.
<br>
<br>
We still need to have the getNextFile as it made the
resolver more flavor
<br>
neutral.
<br>
<br>
Shankar easwaran
<br>
<br>
<br>
<br>
On Oct 21, 2014, at 17:58, Rui Ueyama
<a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<br>
On Tue, Oct 21, 2014 at 3:41 PM, Michael Spencer
<a class="moz-txt-link-rfc2396E" href="mailto:bigcheesegs@gmail.com"><bigcheesegs@gmail.com></a>
<br>
wrote:
<br>
<br>
<blockquote type="cite">On Mon, Oct 20, 2014 at 9:21 PM, Rui
Ueyama <a class="moz-txt-link-rfc2396E" href="mailto:ruiu@google.com"><ruiu@google.com></a> wrote:
<br>
<blockquote type="cite">I'm not still sure whether or not
"InputGraph" thing was after all the
<br>
</blockquote>
right way to abstract the input file list. That often
makes easy things
<br>
hard. In this case, sorting a list of input files became
that hard. When I
<br>
proposed the idea I was thinking that would make this kind
of things easier
<br>
to handle. I don't think the current shape of the API is
not desirable at
<br>
least. Or the fundamental idea was not very good.
<br>
<blockquote type="cite">One thing we probably should
consider is, in my opinion, we are
<br>
</blockquote>
pushing too hard to separate all ports. We are trying to
write any
<br>
architecture-dependent code into architecture-specific
file. But the
<br>
natural border of API doesn't always fit to the
architecture
<br>
dependent/independent border. We should probably relax
that constraint a
<br>
bit where that makes sense and write code that's short and
easier to
<br>
understand.
<br>
<br>
I agree. The current InputGraph code makes modifying the
input very
<br>
difficult to do. Part of the reason for the model in the
first place
<br>
was to make this easy. I'm currently working on adding
#pragma lib
<br>
like support for ELF (PS4 specific stuff) and have found
it basically
<br>
impossible to add a new input file in the correct position
in the
<br>
graph with the current model.
<br>
<br>
</blockquote>
To deal with the issue of the InputGraph I have a few ugly
hacks in
<br>
PECOFF/LinkerGeneratedSymbolFile.h. There are a few virtual
.a files there
<br>
that generate magical symbols on the fly to control the core
linker in an
<br>
obscure way. These hacks need to be removed, but because the
InputFile API
<br>
exists in between, it's impossible to do.
<br>
<br>
The InputGraph API design needs to be revisited and probably
be rewritten.
<br>
<br>
<br>
</blockquote>
</blockquote>
<br>
<br>
_______________________________________________
<br>
llvm-commits mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a>
<br>
</blockquote>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
llvm-commits mailing list
<a class="moz-txt-link-abbreviated" href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>
<a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation</pre>
</body>
</html>