[LLVMdev] [Proposal] Parallelize post-IPO stage.

[Shuxin Yang]

On 7/17/13 4:53 PM, Nick Kledzik wrote:
>
> On Jul 17, 2013, at 4:29 PM, Shuxin Yang <shuxin.llvm at gmail.com 
> <mailto:shuxin.llvm at gmail.com>> wrote:
>
>> On 7/17/13 4:12 PM, Nick Kledzik wrote:
>>> On Jul 14, 2013, at 7:07 PM, Andrew Trick <atrick at apple.com 
>>> <mailto:atrick at apple.com>> wrote:
>>>> The partitioning should be deterministic. It’s just that the linker 
>>>> output now depends on the partitioning heuristics. As long that 
>>>> decision is based on the input (not the host system), then it still 
>>>> meets Eric’s requirements. I just think it’s unfortunate that 
>>>> post-IPO partitioning (or more generally, parallel codegen) affects 
>>>> the output, but may be hard to avoid. It would be nice to be able 
>>>> to tune the partitioning for compile time without worrying about 
>>>> code quality.
>>> I also want to chime in on the importance of stable binary outputs. 
>>>  And not just same compiler and same sources produces same binary, 
>>> but that minor changes to either should cause minor changes to the 
>>> output binary.   For software updates, Apple updater tries to 
>>> download only the delta to the binaries, so we want those to be as 
>>> small as possible.  In addition, it often happens late in an OS 
>>> release cycle that some critical bug is found and the fix is in the 
>>> compiler.  To qualify it, we rebuild the whole OS with the new 
>>> compiler, then compare all the binaries in the OS, making sure only 
>>> things related to the bug are changed.
>>>
>> We can view partitioning as a "transformation".  Unless the 
>> transformation is absolutely no-op,
>> it will change something.   If we care the consistency in binaries, 
>> we either consistently use partition
>> or consistently not use partition.
> But doesn’t "consistently not use partition” mean “don’t use the 
> optimization you are working on”?
Yes

>   Isn’t there someone to get the same output no matter how it is 
> partitioned?
No.  Just like "cc --without-inline" and "cc --with-inline" yields diff 
binaries.


>
>
>>
>>>
>>>>  The compiler used to generate a single object file from the merged
>>>> IR, now it will generate multiple of them, one for each partition.
>>> I have not studied the MC interface, but why does each partition 
>>> need to generate a separate object file?  Why can’t the first 
>>> partition done create an object file, and as other partitions 
>>> finish, they just append to that object file?
>>
>> We could append the object files as an alternative.
>> However, how do we know the /path/to/ld from the existing interface ABIs?
>> How do we know the flags feed to the ld (more often than not, "-r" 
>> alone is enough,
>> but some linkers may need more).
>>
>> In my rudimentary implement, I hack by hardcoding to /usr/bin/ld.
>>
>> I think adding object one by one back to the linker is better as the 
>> linker already have
>> enough information.
> I think you missed my point, or you are really thinking from the 
> multi-process point of view.   In LLVM there is an MCWriter used to 
> produce object files.   Your model is that if there are three 
> partitions, then there will be three MCWriter objects, each producing 
> an object file.  What I am saying is to have only one MCWriter object 
> and have all three partitions stream their content out through the one 
> MCWriter, producing one object file.
>
No, I don't want to sync all threads at this point. It make things all 
the more complex.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130717/c8834b57/attachment.html>