[llvm-dev] Commit module to Git after each Pass

Wed Mar 21 03:08:03 PDT 2018

On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote:
 > git-commit-after-all solution has one serious issue - it has a 
hardcoded git handling which
 > makes it look problematic from many angles (picking a proper git,
 > selecting exact way of storing information, creating repository, 
replacing the file etc etc).
 >
 > Just dumping information in a way that allows easy subsequent machine 
processing
 > seems to be a more flexible, less cluttered and overall clean 
solution that allows to avoid
 > making any of "user interface" decisions mentioned above.
 >
 > We need to understand why git-commit-after-all works faster than 
print-after-all.
Made an interesting experiment today and extended your 
git-commit-after-all to avoid issuing
any git commands if git-repo starts with "/dev/".

With git-repo=/dev/stderr it becomes functionally equivalent to 
print-after-all+print-module-scope,
dumping module into stderr after each pass.

On my testcase:

# first normal git-commit-after-all execution
] rm -rf test-git; time $RR/bin/opt -O1 some-ir.ll -disable-output 
-git-commit-after-all -git-repo=./test-git

real    0m7.172s
user    0m6.303s
sys     0m0.902s
# then "printing" git-commit-after-all execution
] time $RR/bin/opt -O1 some-ir.ll -disable-output -git-commit-after-all 
-git-repo=/dev/stderr 2>&1 | grep -c '^; ModuleID'
615

real    0m2.893s
user    0m2.859s
sys     0m0.356s
# and finally print-after-all
] time $RR/bin/opt -O1 some-ir.ll -disable-output -print-after-all 
-print-module-scope 2>&1 | grep -c "^; ModuleID"
526

real    2m8.024s
user    0m55.933s
sys     3m19.253s
]
Ugh... 60x???
Now, I'm set to analyze this astonishing difference that threatens my 
sanity (while I'm still sane ... hopefully).

regards,
   Fedor.
PS btw, I checked /dev/null - and it works faster than /dev/stderr as 
expected :)

 > I dont believe in magic... yet :)
 >
 > And, btw, thanks for both the idea and the patch.
 >
 > regards,
 >   Fedor.
 >
 > On 03/16/2018 12:03 AM, Alexandre Isoard wrote:
 >> If this is faster than -print-after-all we may actually consider 
pushing that in the code base then? (after diligent code review of course)
 >>
 >> Note that it uses the same printing method as -print-after-all:
 >> - create a pass of the same pass kind as the pass we just ran
 >> - use Module::print(raw_ostream) to print (except -print-after-all 
only print the concerned part and into stdout)
 >>
 >> If there is improvement to be done to print-after-all it might also 
improve git-commit-after-all. (unless that only improve speed when 
printing constructs smaller than module)
 >>
 >> In any case, it is, to me, much more usable (and extensible) than 
-print-after-all. But requires git to be in PATH (I'm curious if that 
works on Windows).
 >>
 >> On Thu, Mar 15, 2018 at 1:35 PM, Daniel Sanders 
<daniel_l_sanders at apple.com> wrote:
 >>
 >>     Does https://reviews.llvm.org/D44132 help at all?
 >>
 >>
 >>>     On 15 Mar 2018, at 09:16, Philip Reames via llvm-dev 
<llvm-dev at lists.llvm.org> wrote:
 >>>
 >>>     The most likely answer is that the printer used by 
print-after-all is slow.  I know there were some changes made around 
passing in some form of state cache (metadata related?) and that running 
printers without doing so work, but are dog slow.  I suspect the 
print-after-all support was never updated.  Look at what we do for the 
normal IR emission "-S" and see if print-after-all is out of sync.
 >>>
 >>>     Philip
 >>>
 >>>     On 03/15/2018 08:45 AM, Alexandre Isoard via llvm-dev wrote:
 >>>>     Huh. Great! 😁
 >>>>
 >>>>     I don't believe my poor excuse from earlier (else we should 
map all pipes into files!), but I'm curious why we spend less time in 
system mode when going through file than pipe. Maybe /dev/null is not as 
efficient as we might think? I can't believe I'm saying that...
 >>>>
 >>>>     On Thu, Mar 15, 2018, 08:25 Fedor Sergeev 
<fedor.sergeev at azul.com> wrote:
 >>>>
 >>>>         Well, git by itself is so focused on performance, so its 
not surprising
 >>>>         to me that even using git add/git commit does not cause
 >>>>         performance penalties.
 >>>>
 >>>>
 >>>>     Sure, but still, I write more stuff (entire module) into a 
slower destination (file). Even ignoring git execution time it's counter 
intuitive.
 >>>>
 >>>>     The only difference is that while I write more, it overwrite 
itself continuously, instead of being a long linear steam. I was 
thinking of mmap the file instead of going through our raw_stream, but 
maybe that's unnecessary then...
 >>>>
 >>>>
 >>>>
 >>>>     _______________________________________________
 >>>>     LLVM Developers mailing list
 >>>>     llvm-dev at lists.llvm.org
 >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
 >>>
 >>>     _______________________________________________
 >>>     LLVM Developers mailing list
 >>>     llvm-dev at lists.llvm.org
 >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
 >>
 >>
 >>
 >>
 >> --
 >> Alexandre Isoard
 >
 >
 >
 > _______________________________________________
 > LLVM Developers mailing list
 > llvm-dev at lists.llvm.org
 > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev