[PATCH] sectiongroup support

[Nick Kledzik]

On Mar 15, 2014, at 10:02 PM, Shankar Easwaran <shankare at codeaurora.org> wrote:

> Hi Nick,
> 
> This is the overall design that I am working on to support reading/resolving section groups in the YAML reader.
> 
> a) Read all atoms
> b) any atoms with group-parent references are moved to AtomList<lld::DefinedAtom> _groupChildAtoms;
> c) There would also be a map _groupChildMap that has a key from groupChild to groupParent, this is to support (d)
> d) If there is a reference to a groupChild from another child(whose parent is different from the previous group child), the groupChild reference is changed to refer to a undefined atom for the groupChild
> e) if there is a reference to a groupchild that doesnot have a group-parent reference, and if the reference is not to a atom that is undefined, create an undefinedatom and point the group child.
It is not clear if you mean the ELF Reader will be doing this or the SymbolTable.  I think any creation of UndefinedAtoms should be done in the ELF Reader to properly present the ELF file as atoms.  The SymbolTable/Resolver just coalesce atoms and bind references.


> We could also do something where all groupchild references refer to undefined atoms.
> 
> There seems to be a caveat even after everything looks sane.
> 
> a) There is a funny case of what do we do when the group child is a local symbol ? So if there is a reference to a group child whose scope is local, we cannot create a undefined reference to the group child right ?
From <http://docs.oracle.com/cd/E23824_01/html/819-0690/chapter7-26.html>
References to the sections comprising a group from sections outside of the group must be made through symbol table entries with STB_GLOBAL or STB_WEAK binding and section indexSHN_UNDEF. A definition of the same symbol in the object containing the reference must have a separate symbol table entry from the reference. Sections outside of the group can not reference symbols with STB_LOCAL binding for addresses that are contained in the group's sections, including symbols with type STT_SECTION.

I read that as references to local symbol in a group can only come from other symbols in the group.  It also sounds like the ELF file already have undefined symbols to reference definitions in a group.  


> b) The problem is we call the RoundTripYAML reader/writer and the RoundTripNative reader/writer, which happens to be called after the resolver is called, which would create undefined references to atoms!.
The issue is that to support easy replacement, groups are purposely unresolved (non-canonical) in .o files.  Normally, that is fine because they will be processed through the Resolver which will replace any undefines yielding a canonical atom graph.  But RoundTripYAML (and maybe RoundTripNative) could regenerate the unresolved state which will not be sent through the resolver again, so the Writer or other Passes will see a non-canonical atom graph.  

The same issue arises if the linker is being using to merge .o files (-r mode).  The resolver will produce a canonical graph (no extra undefines), but the ELF Writer will want them.

I think the solution:
1) keep the existing canonical form for an atom graph that Passes process
2) make the ELF Writer smart that when it writes relocatable object files, to synthesize undefined symbols for references into group children
3) Have the YAML Reader be dumb and not synthesize any undefines.  Test cases for groups that want undefines will need to explicitly have them in the yaml.  This also enables you to write test cases that test the error case where an indirection though an undefine is not used.


-Nick


> I am concerned on just the above cases ?
> 
> Do we want to have a different mode in the YAML reader when called from RoundTripYAMLPass not to create undefined atoms ? 
> 
> Am I missing something ?
> 
> Thanks
> 
> Shankar Easwaran
> 
> On 3/14/2014 8:29 PM, Nick Kledzik wrote:
>> On Mar 14, 2014, at 4:37 PM, Shankar Easwaran <shankare at codeaurora.org> wrote:
>> 
>>> On 3/14/2014 5:52 PM, Nick Kledzik wrote:
>>>> On Mar 13, 2014, at 8:24 PM, Shankar Kalpathi Easwaran <shankarke at gmail.com> wrote:
>>>>>  I dont think the multithreaded model of processing all the object files first is going to work with the ELF linker, as resolution happens in the way input files are processed from command line.
>>>>> 
>>>>>  This is what I see with the GNU linker :-
>>>>> 
>>>>>  a) The groups are being resolved as and when input files are processed by the resolver. We cannot delay processing the group members after the resolver has processed all files.
>>>>>  b) Consider the below example :-
>>>> ...
>>>> 
>>>>>  With the command line ld foo.o bar.o libarchive.a, when symbols from bar.o are processed, **global-f3 becomes a undefined symbol** and pulls the global-f3 from libarchive.a
>>>>> 
>>>>>  How do we handle all these cases with lld ?
>>>>> 
>>>>> http://llvm-reviews.chandlerc.com/D2961
>>>> Given that the ELF linker needs those semantics, it seems that the Resolver and symbol table building must be done serially, in command line order.  The object files could still be parsed in parallel, but the results would need to be buffered and added to the symbol table in command line order.  Given that, here are two possible algorithms:
>>>> 
>>>> 1) Groups are like static libraries
>>>> a) group child atoms are not returned in the defined() atoms collection, but group parents are.
>>>> b) files are added to the symbol table in command line order
>>>> c) when a group parent atom is added to the symbol table, its name is checked to see if that group name has been seen yet by the symbol table.  If not, the group name is added and all its children are also added.  If the group name has already been seen, then the group’s child remain unseen by the symbol table.
>>>> Problem: uses of any atom in a group from outside the group would require the ELF reader to synthesized UndefinedAtom and have that be referenced. In a way this is good because it enforces that groups can be removed.
>>> Thanks for the great suggestion. We could follow the option (1) above, as the group childs are not seen by the rest of the linker until the group is added.
>>> 
>>> Are you saying that the group childs are internal to the lld::File, and can be only referenced from the group-child references in the parent atom (contentType:kindGroupComdat) ?
>> Without adding some other mechanism for getting a list of atoms, yes, that is how to do it.
>> 
>>> To support groups in the core/yaml reader, the user has to be explicit in creating undefined/defined atoms for the group child members ?
>> I supposed you could have the yaml reader detect and automatically create the UndefinedAtoms.  I’m not sure which is easier.
>> 
>> 
>> Also, this is going to require making lots of error laden test cases in ELF assembly and see how the existing ELF linkers handle them.  For instance if group g1 contains a function that calls foo, and that is the only call to foo in the object file, when that file is loaded, if g1 is not used (because another group named g1 already exists), will the undef for foo get added to the global symbol table?  If not, then somehow undefined symbols also need to be associated with groups.
>> 
>> -Nick
>> 
>> 
>> 
> 
> 
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140317/d33db066/attachment.html>