[Patch][Review Request][llvm-link] Fix One Performance Bug

[Bill Wendling]

Hi Wan Xiaofei,

Hooray! You identified one of the major slowdowns in the bitcode linking code. I noticed this awhile ago, but haven't had time to look at it. I'm impressed by your speedups! I'll take a closer look at your code.

For what it's worth, I also think that it would be better not to have bufNamesStructTypesSet as part of Module.

Thanks again!

-bw

On Apr 29, 2013, at 7:46 AM, "Wan, Xiaofei" <xiaofei.wan at intel.com> wrote:

> Stefanus:
> 
> Thanks for your suggestion, it is better to limit bufNamesStructTypesSet within llvm-link, this is what I planned to do at the beginning; I just worry about it will affect other component where ModuleLinker is used
> 
> Thanks
> Wan Xiaofei
> 
> -----Original Message-----
> From: Du Toit, Stefanus 
> Sent: Monday, April 29, 2013 9:44 AM
> To: Du Toit, Stefanus; Wan, Xiaofei; llvm-commits at cs.uiuc.edu
> Subject: Re: [Patch][Review Request][llvm-link] Fix One Performance Bug
> 
> On 2013-04-28 9:27 PM, "Du Toit, Stefanus" <stefanus.du.toit at intel.com>
> wrote:
> 
>> I don't think it makes sense to add bufNamesStructTypesSet to Module. 
>> Can you keep this in the linker code instead, and maintain an mapping 
>> from
>> Module* to the corresponding set there?
> 
> To be more specific, I think the allocation of this data structure belongs in llvm-link itself.
> 
> Since that code already has a single destination module ("Composite"), that would also be a good place to allocate this data structure. You could then pass a pointer to it as an optional parameter to ModuleLinker's constructor.
> 
> That way, you avoid having to do any special book-keeping (e.g. a map from
> Module* to the set as I suggested in my first message) and don't have to worry about what would happen if the Module got deleted in that case, etc.
> 
> Stefanus
> 
>> From: <Wan>, Xiaofei 
>> <xiaofei.wan at intel.com<mailto:xiaofei.wan at intel.com>>
>> Date: Sunday, 28 April, 2013 1:43 AM
>> To: "llvm-commits at cs.uiuc.edu<mailto:llvm-commits at cs.uiuc.edu>"
>> <llvm-commits at cs.uiuc.edu<mailto:llvm-commits at cs.uiuc.edu>>
>> Subject: [Patch][Review Request][llvm-link] Fix One Performance Bug
>> 
>> Background:
>> Our business need to link many BC files together to generate a final 
>> big BC file; we noticed that the linking time increase 
>> significantly(more than linearly) with the increase of the BC file 
>> number, our biggest case (with ~8000 BC files ) takes > 1 hour time on 
>> XEON server which is unbearable to users.
>> 
>> Profiling and Analysis:
>> 
>> a.       From profiling data, it was observed that TypeFinder() consumes
>> 98% of total time; TypeFinder() is not a key activity in llvm-link, it 
>> is just used to filter out all ³Named Structure Types² in a module
>> 
>> b.       In current llvm-link, when linking all BC files, the BC file is
>> linked one by one; for example, link bc1, bc2, bc3, bc4, bc5, bcN, the 
>> workflow is that, link bc1 & bc2 to bc12, then link with bc3 to bc123, 
>> then link with bc4 to bc1234, Š, finally bc12345ŠN.
>> 
>> The ³Named Structure Types² in destination module is calculated in each 
>> linking; with the size increase of destination module, TypeFinder() 
>> will consume more and more time, this explains why the linking time 
>> increase is more than linearly
>> 
>> Solution & Fix & Result:
>> 
>> a.       ³Named Structure Types² can be maintained in an incremental way
>> in destination module, when linking a new module, just need to add new 
>> ³Named Structure Types² into destination module to keep it most 
>> up-to-date
>> 
>> b.       The fix is very small, just add a collection in module to keep
>> all ³Named Structure Types² in destination module for link. See 
>> attachment for details.
>> 
>> c.       After this fix:
>> 
>> For our biggest case, the linking time decrease from 70 minutes to 35 
>> seconds.
>> 
>> ForXalan in SPEC2006, the linking time decrease from 30 seconds to 2 
>> second
>> 
>> Please review, thanks!
>> 
>> --
>> Best Regards
>> Wan Xiaofei (xiaofei.wan at intel.com<mailto:xiaofei.wan at intel.com>)
>> Intel Corporation, Shanghai, China
>> MCG, Android Runtime & Compiler Team
>> 
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits