[PATCH] Do not use layout-before to layout atoms.

[Shankar Easwaram]

Sent from my iPhone

> On Mar 26, 2014, at 20:03, Rui Ueyama <ruiu at google.com> wrote:
> 
>> On Wed, Mar 26, 2014 at 5:42 PM, Shankar Easwaran <shankare at codeaurora.org> wrote:
>>> On 3/26/2014 7:19 PM, Rui Ueyama wrote:
>>> On Wed, Mar 26, 2014 at 4:53 PM, Shankar Easwaran <shankare at codeaurora.org> wrote:
>>>>> On 3/26/2014 6:25 PM, Rui Ueyama wrote:
>>>>> On Wed, Mar 26, 2014 at 4:11 PM, Shankar Easwaran
>>>>> <shankare at codeaurora.org>wrote:
>>>>> 
>>>>>>>  On 3/26/2014 5:55 PM, Nick Kledzik wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>  On Mar 26, 2014, at 3:37 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>> 
>>>>>>>   On Wed, Mar 26, 2014 at 3:09 PM, Nick Kledzik <kledzik at apple.com> wrote:
>>>>>>> 
>>>>>>>>  On Mar 26, 2014, at 2:56 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>>> 
>>>>>>>>   On Wed, Mar 26, 2014 at 2:51 PM, Nick Kledzik <kledzik at apple.com>wrote:
>>>>>>>> 
>>>>>>>>>  On Mar 26, 2014, at 2:44 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>>>> 
>>>>>>>>>   On Wed, Mar 26, 2014 at 2:23 PM, kledzik at apple.com <kledzik at apple.com>wrote:
>>>>>>>>> 
>>>>>>>>>   Given your description of COFF, I'm not sure follow-on
>>>>>>>>> atom/references is the right model.
>>>>>>>>> 
>>>>>>>>>   We probably want a way to order sections.  For instance, the darwin
>>>>>>>>> linker automatically arranges sections for best performance.
>>>>>>>>> 
>>>>>>>>  Yes, follow-on atom/references model is not probably the best model
>>>>>>>> for COFF. It's too fine grained. The unit of layout is not an atom (symbol)
>>>>>>>> but a section. We never want to rearrange or dead-strip each atom. AFAIK so
>>>>>>>> is true for ELF and Mach-O.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>  The darwin linker very much dead strips and re-orders mach-o at the
>>>>>>>> atom level.
>>>>>>>> 
>>>>>>>  Maybe silly question, but how are dependencies between symbols (atoms)
>>>>>>> represented in Mach-O? I mean, if function A in .text have a relative call
>>>>>>> instructions to function B in the same .text section, both functions needs
>>>>>>> to be in the final executable keeping the same offset.
>>>>>>> 
>>>>>>>  The call instruction has a relocation on it, so if the target moves
>>>>>>> (relatively), the linker updates the call instruction.
>>>>>>> 
>>>>>>>  In more detail, the linker parses the call instruction and relocation
>>>>>>> and adds to A, a Reference to B.  Then everything proceeds as an atom
>>>>>>> graph, including dead stripping and re-ordering.  In the writer, after
>>>>>>> atoms that remain are assigned addresses, the writer sees the reference in
>>>>>>> A to B, it knows the address of both, so it fixes up the call instruction
>>>>>>> given those final addresses.
>>>>>>> 
>>>>>>  Thank you for the explanation. That sounds like a different semantics
>>>>>> than COFF indeed. In COFF we don't usually have relocations within a
>>>>>> section, so a section is a basic unit and we can't reorder or remove data
>>>>>> in it. If you want to reorder and dead-strip each function, you have to
>>>>>> specify /Gy (equivalent to -ffunction-section) to put each function in its
>>>>>> own section.
>>>>>> 
>>>>>>  Do you know if we really use layout-after's in ELF to order atoms? It
>>>>>> seems to me that the default sorting function orders them correctly,
>>>>>> according to atom's ordinal, file ordinal, etc. Is there any case that we
>>>>>> are doing more than that using layout-after edges?
>>>>>> 
>>>>>> You’ll have to ask Shankar what problem he was trying to solve with the
>>>>>> layout before/after stuff.
>>>>>> 
>>>>>>   ELF does not use layout-before references, it uses only layout-after
>>>>>> and in-group references. The reason the ELF linker tried to do this was
>>>>>> 
>>>>>> a) Prevent reordering of atoms and maintain section relationship. All
>>>>>> atoms in a section have to be tied together. Anything that tries to reorder
>>>>>> has to reorder the whole section.
>>>>>> b) This also would clearly model linker scripts in the future that moving
>>>>>> a function would make sure the section that contains the function is moved,
>>>>>> and not the individual function.
>>>>>> 
>>>>>> This was earlier discussed in the thread :
>>>>>> http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20121022/154212.html
>>>>>> 
>>>>>> I would like to have the kindLayoutBefore/kindLayoutAfter/kindInGroup
>>>>>> references to exist and do the functionalities that it is trying to
>>>>>> accomplish.
>>>>>> 
>>>>>> This could be used when we want to implement the order file in the near
>>>>>> future.
>>>>>> 
>>>>> Is there anything that is presently depending on layout-after and in-group,
>>>>> which does not work with the default ordering function, compareAtomsSub()?
>>>>> I don't intend to say that that wouldn't be needed for ELF, but just trying
>>>>> to understand.
>>>>> As to the feature set that LayoutPass provides, we shouldn't keep this
>>>>> complexity as is as I repeatedly stated, because as I sayd multiple times
>>>>> it seems that this level of complexity is not (and will not) needed.
>>>> Tell me another way to accomplish the same thing that I would want to order atoms with a linker script ?
>>>> 
>>>> How can I make sure the section that contains the atoms                 are moved as a group without this change ?
>>> 
>>> That's what I'm asking. If some feature is needed for some ELF linker script feature, we need to implement or keep it, so I wanted to know actual examples. Also, what I'm asking in the first sentence is about the current implementation of ELF support in LLD.
>> The current functionality thats tested in the tree, uses kindLayoutAfter/kindInGroup only to model ELF in lld. Its only used to model ELF sections.
>> 
>> I use it in environments internally that reads gprof data and see if it arranges atoms appropriately, which make use of this functionality.
> 
> If you need some feature for your internal organization use only and do not upstream, I'd think that use cannot justify keeping that functionality in upstream in general.
>  
As I said before this functionality is critical for linker scripts and ELF.
>>>>> Specifically, layout-before relationship is not really needed for ay
>>>>> occasion to layout atoms -- in all cases when you want to layout atom A
>>>>> before B, you can just add "layout-after to A" to B instead. Dead-stripping
>>>>> path would still need doubly-linked graph, but it does not mean that
>>>>> layout-pass needs to use it too.
>>>> I think I have said many times as well, again ELF uses only layout-after and in-group.
>>>> 
>>>> You cannot model weak symbols with just layout-after reference alone. You cannot have a Layout After reference to a weak symbol, as when the weak symbol could be overridden.
>>>> 
>>>> To make function alias symbols, ELF needs kindLayoutBefore reference, so that the linker can easily setup an alias to a function(the +afs option).
>>>> 
>>>> The alias symbols are again zero sized symbols that are defined through the command-line.
>>> 
>>> I don't see the reason why you can't use kindLayoutAfter instead of kindLayoutBefore to enforce layout in that case. In order to create a layout-before edge from atom A to B, you need to have references both to A and B. So adding "layout-before A" edge to B is as easy as adding "layout-after B" edge to B.
>>> 
>>> For example, if you already have A -> B ("->" means "layout-after") and want to add X (zero size atom) as an alias for A, you can insert X after A, so that the resulting graph looks like A -> X -> B. It can also be represented as A -> B and B => X ("=>" means "layout-before"). My point is that we can choose either of two, but the former is simpler.
>>> 
>>> I'm not suggesting we remove in-group or layout-after. I'm against keep supporting layout-before in the layout pass.
>> Its much easier to do with kindLayoutBefore. Its easy to handle if its an alias to a section symbol, which exists as the first symbol in the section. 
>> 
>> The only thing I wanted to say is not to remove/fix the infiniteloop for kindLayoutBefore functionality as things become much easier.
>> 
>> Just mentioning, In your example, A->X->B produces an alias to B (X and B would share the same address), we need X->A->B.
> 
> It's much easier to do with layout-before, because we have a complex piece of code in LayoutPass.cpp for that feature. That simplicity of use is not free.
> 
> Layout-before is overly flexible mechanism, and its feature overlaps too much with layout-after. If you want to use layout-before for symbol aliases, you could define new kind of relationship between symbols, say "alias" relationship, to layout them at the same offset. That would be much easier to implement in the layout pass, and it should be as easy to use as layout-before. I don't still see the reason we really need layout-before and no other mechanisms.
> 

If you do this in the elf writer that information would not be captured in the YAML or the native representation.

I am not convinced, as long as the functionality is not exercised by having a flag, I don't see why it impacts you at this time.


>> <--snip from earlier message from Nick-->
>> 
>>   In terms of improving performance of this pass, we could add a flag to the MergedFile that indicates if any Atom in it has a follow-on (layout before or layout after reference).  If there are none (which is common for COFF and mach-o), then all the _followOnRoots and _followOnNexts set up can be skipped.
>> <--end snip from earlier message from Nick-->
>> 
>> We should plan to do the above, as the suggestion from Nick should greatly improve performance, which is the most concerning point.
>> 
>> PS: Overloading kindLayoutBefore just for garbage collection, is something that you may want to think of changing in your patch (If you intend to use LayoutBefore references).
>> 
>> 
>> Thanks
>> 
>> Shankar Easwaran
>> -- 
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140326/7333ee1d/attachment.html>