[llvm-dev] Commit module to Git after each Pass

Thu Mar 15 07:47:31 PDT 2018

Hmm...

I tried Alexandre's fix from D44244 and surprisingly it appears that 
just using -print-module-scope w/o
any additional git actions is waaaay slower on my testcase than 
-git-commit-module-all.

Hell, even a plan -print-after-all is slower:

  ] time R/bin/opt -O3 some-ir.ll -disable-output -git-commit-after-all 
2>/dev/null
real    0m8.041s
user    0m7.133s
sys     0m0.936s
] time R/bin/opt -O3 some-ir.ll -disable-output -print-after-all 2>/dev/null

real    0m13.575s
user    0m6.179s
sys     0m7.394s

I cant really explain that...

regards,
   Fedor.

On 03/15/2018 04:30 PM, Fedor Sergeev via llvm-dev wrote:
>
>
> On 03/15/2018 01:32 PM, Fedor Sergeev via llvm-dev wrote:
>> For this to be really usable in this setup we need additionally to:
>>   - extend -print-module-scope to cover basic block passes
>>   - introduce a clear way to separate module IRs as those are being 
>> printed by -print-after-all
>>
>> But yes, it should work, and a wrapper that pipes to git fast-import 
>> seems to be the best way to handle it.
> A simple 20-lines perl script does the trick pretty easily: 
> https://pastebin.com/4J0b5Tr8
>
> (this assumes my local modification to introduce the *** END OF ** IR 
> DUMP marked at the end of -print-module-scope's IR module dump)
>
> ] git init
> ] RA/bin/opt -O3 some-ir.ll -disable-output -print-after-all 
> -print-module-scope 2>&1 | filter-LLVM-ir-print.pl | git fast-import 
> --done --date-format=now
> ....
>
> Majority of time is spent to actually print the IR (~2m for my testcase).
> Fast-import takes just a second.
>
> regards,
>   Fedor.
>
>>
>> regards,
>>   Fedor.
>>
>> On 03/15/2018 12:31 AM, Daniel Neilson via llvm-dev wrote:
>>> The print-module-after-all type of option exists in upstream:
>>>   -print-module-scope                                     - When 
>>> printing IR for print-[before|after]{-all} always print a module IR
>>>
>>> commit 7d160f714357f6784ead669ce516e94991c12e5a
>>> Author: Fedor Sergeev <fedor.sergeev at azul.com 
>>> <mailto:fedor.sergeev at azul.com>>
>>> Date:   Fri Dec 1 17:42:46 2017 +0000
>>>
>>>     IR printing improvement for function passes - introducing 
>>> -print-module-scope
>>>
>>>
>>>     Summary:
>>>     When debugging function passes it happens to be rather useful to 
>>> dump
>>>     the whole module before the transformation and then use this dump
>>>     to analyze this single transformation by running it separately
>>>     on that particular module state.
>>>
>>>
>>>     Introducing
>>>         -print-module-scope
>>>     debugging option that forces all the function-level IR dumps
>>>     to become whole-module dumps.
>>>
>>>
>>>     This option builds on top of normal dumping controls like
>>>        -print-before/after
>>>        -filter-print-funcs
>>>
>>>
>>>     The plan is to eventually extend this option to cover other 
>>> local passes
>>>     (at least loop passes) but that should go as a separate change.
>>>
>>>
>>> Loop passes here:
>>> commit 5608259c999fb77c5d6093895696f4daebe6b8cd
>>> Author: Fedor Sergeev <fedor.sergeev at azul.com 
>>> <mailto:fedor.sergeev at azul.com>>
>>> Date:   Fri Dec 1 18:33:58 2017 +0000
>>>
>>>     IR printing improvement for loop passes - handle 
>>> -print-module-scope
>>>
>>>
>>> -Daniel
>>>
>>>
>>>> On Mar 14, 2018, at 3:51 PM, Philip Reames via llvm-dev 
>>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>>
>>>> This is interesting, and might be useful.  I don't know that this 
>>>> is broadly useful enough for upstream inclusion, but if you could 
>>>> post this to github somewhere, I might play with it.
>>>>
>>>> There might also be room to factor out common functionality. We've 
>>>> also run into the need to print whole-module instead of containing 
>>>> construct (i.e. this loop).  If we added upstream support for 
>>>> something along the lines of -print-module-after-all, building the 
>>>> git history could easily be done as a post processing step.
>>>>
>>>> Philip
>>>>
>>>>
>>>> On 03/06/2018 10:43 AM, Alexandre Isoard via llvm-dev wrote:
>>>>> Hello,
>>>>>
>>>>> I had a stupid idea recently that turned out not so stupid after 
>>>>> all. I wanted to be able to "see" an entire pass pipeline in 
>>>>> action to find unnecessary transformations and/or missed 
>>>>> opportunities and generally improve the debug-ability of LLVM.
>>>>>
>>>>> So as the title suggest, I implemented an equivalent of 
>>>>> "-print-after-all" but instead of printing into stdout I dump into 
>>>>> a file that get commit into a temporary git. There are some quirks 
>>>>> with it but it's working and is actually awesome. For example, at 
>>>>> first sight, I see multiple time lcssa and instcombine cancelling 
>>>>> each other's work.
>>>>>
>>>>> Of course, that has a big impact on compile time when enabled, but 
>>>>> that's still practical (git being quite good at its job) when 
>>>>> debugging.
>>>>>
>>>>> There are improvement I can make, but would you guys be interested 
>>>>> in such feature?
>>>>>
>>>>> -- 
>>>>> *Alexandre Isoard*
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev