<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Nov 21, 2012, at 8:55 AM, Relph, Richard wrote:</div><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
AMD would like to add new functionality to ranlib (and later ar and nm) and to the bits of LLVM Core that read (and later write) archives.
<div>Herewith a terse summary of the change, which we want to improve support of OpenCL for multiple GPUs in a single run-time.<br>
<br>
<div>Conceptually, a serialized archive is really 2 pieces: a few header members and a set of normal file members. There are no constraints on the normal members in the 'pure' archive format. They could be text files, pictures, or, as we're all familiar with,
object modules. Most object file archives are "libraries" and the have a special header member that is a global symbol table, associating global scope names with defining object module members in the archive body.</div>
<div><br>
</div>
We have N very large archives, defining essentially the same set of symbols. Many of the normal file members of each are duplicated in other archives, but not all. The goal is the produce a single "super-archive" that contains 1 copy of each unique object file
member no matter how many archives it is part of, and N symbol table members representing each of the original N archives.</div></div></blockquote><div>Let me see if I understand your need here. You are dynamically generating code and need to link it. The linking step requires some support routines which makes sense to have in a static library. Since this must work on machines not set up with developer tools, you are packaging the static library inside a DLL/DSO. In addition, with all the minor variations of GPUs, having a separate archive for every GPU type would be too large, so you need some way to remove duplicates of support functions.</div><div><br></div><div>If the above summary is close, then here are two other ideas that avoid the need for archive/TOC changes:</div><div>1) Have lots of little archives which removes duplicates. Give each archive a unique name, then have a lookup table which lists which sequence of archives to use for which specific GPU. When linking for a particular GPU, you pass the linker that particular sequence of little archives to search. </div><div>2) Have one giant archive for per GPU family and use name mangling scheme to filter. For instance, the compiler emits references to support routine "foo" as "foo$gpu1". Then you construct the support libraries with aliases for each support function. So a particular implementation of foo in an archive may show up in the symbol table through its aliases "foo$gpu3", "foo$gpu4", "foo$gpu7". When the linker is only looking for "foo$gpu1", it will ignore all other foo implementations and just pick the one aliased to "foo$gpu1".</div><div><br></div><div><br></div><div>BTW, Apple/Darwin has a similar issue with supporting multiple CPUs. Our solution is "fat" archives. The ranlib tool sorts all archive members by cpu, builds "thin" archive libraries for each cpu, then concatenates the thin libraries together with a "fat header" which specifies the file offset and size of each thin archive. The linker, when it comes across a fat static library, looks at the fat header thens seeks along to the contained thin archive relevant to the cpu type being linked. This scheme works well, but it can produce large files because there is no attempt to reduce duplicates.</div><div><br></div><div>-Nick</div><div><br></div><div><br></div><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br>
</div>
<div>The symbol table for each original archive can properly index to the relevant members in the archive, even if other members in the super-archive (not referenced in this particular symbol table, of course) define the same symbols.<br>
<br>
</div>
<div>I've considered 3 approaches to the problem so far. All involve a new archive member type.</div>
<div><br>
</div>
<div>First, a new archive member type "up front" that describes each of the original archives and its symbol table.</div>
<div>Second, a normal/default symbol table member "up front" and a new archive member type that describes alternate symbol tables contained in the archive.</div>
<div>Third, a "hiding" archive member type that is essentially a way to "skip over" additional normal archive file headers to reach the first normal member, which (in all approaches) all archives share.</div>
<div><br>
</div>
<div>The third, I think, requires the least changes to the existing implementation, so I'm leaning towards it. The "hiding" archive member would have the "file name" of the represented archive immediately following the member header, followed by a completely
normal archive representation starting with <span style="font-family: Arial; ">"<!arch>\n" and optionally including an additional "hiding" archive member covering even more hidden archives.</span></div>
<!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<div><br>
</div>
<div>The plan is to extend the Archive class to provide for a way for clients to select a desired archive. I also will enhance ranlib to accept multiple archive names on the command line and produce the "super-archive" from ranlib.<br>
<br>
<div>A further need we have is to serialize the TOCs and the super-archive in a memory image (our archives are embedded in our DLL/SO, not stored separately on disk) and then provide an interface to the relevant LLVM classes (Linker, primarily) for accessing
archives in memory rather than on disk, a feature absent from the current implementation.</div>
<div><br>
</div>
<div>For our purposes, extending the Archive class to support specification of the archive using a memory object instead of a file, recognizing the "hiding" member type, and extending ranlib to produce the new super archives is all we really need.</div>
<div><br>
</div>
<div>Any thoughts or suggestions would be welcome.</div>
<div><br>
</div>
Thanks,<br>
Richard</div>
<div>
<div><br>
</div>
</div>
</div>
_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:LLVMdev@cs.uiuc.edu">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu">http://llvm.cs.uiuc.edu</a><br><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><br></blockquote></div><br></body></html>