<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">FWIW: We could also just have a mode that dumps 1 file per pass. That is enough to make it convenient/easy to run diff between passes.<div class="">(And if you wanted to you could still make a git repository out of it with an external script).<br class=""><div class=""><br class=""></div><div class="">- Matthias<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 14, 2018, at 10:49 AM, Alexandre Isoard via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hello,<div class=""><br class=""></div><div class="">Just an update on that. I am personally using -git-commit-after-all *as-is* extremely frequently (combined with "git filter-branch" and "opt -S -instnamer" it is extremely useful).</div><div class="">I unfortunately won't have time to write a better implementation of that, and I agree "git fast-import" seems the way to go. If anybody is motivated enough to do so, feel free.</div><div class=""><br class=""></div><div class="">Best regard!</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Thu, Mar 22, 2018 at 10:38 AM Reid Kleckner via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Obviously, we do not want all stderr output to be buffered. However, I think it would be great to change Function::print and Module::print to call raw_ostream::SetBuffered / raw_ostream::SetUnbuffered before and after printing. I guess if the original stream was buffered we don't want to mark it unbuffered, so we may need to tweak the raw_ostream interface. Looks easy, though.</div><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Thu, Mar 22, 2018 at 8:06 AM Fedor Sergeev via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Oh, well... as usually the answer appears to be pretty obvious.<br class="">
99% of the time is spent inside the plain write.<br class="">
<br class="">
-print-after-all prints into llvm::errs(), which is an *unbuffered*<br class="">
raw_fd_stream.<br class="">
And -git-commit-after-all opens a *buffered* raw_fd_stream.<br class="">
<br class="">
As soon as I hacked -print-after-all to use a buffered stream to stderr<br class="">
performance went<br class="">
up to the normal expected values:<br class="">
<br class="">
] time bin/opt -O1 big-ir.ll -disable-output -print-after-all<br class="">
-print-module-scope 2>&1 | grep -c "^; ModuleID"<br class="">
526<br class="">
<br class="">
real 0m2.363s<br class="">
user 0m2.373s<br class="">
sys 0m0.271s<br class="">
]<br class="">
<br class="">
So, the morale of this story is - we should not be printing module IR<br class="">
into dbgs/errs().<br class="">
<br class="">
And then the idea of streaming IR module dumps into a buffered stream<br class="">
and then postprocessing seems<br class="">
to be a right one.<br class="">
<br class="">
regards,<br class="">
Fedor.<br class="">
<br class="">
On 03/21/2018 01:08 PM, Fedor Sergeev via llvm-dev wrote:<br class="">
> On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote:<br class="">
> > git-commit-after-all solution has one serious issue - it has a<br class="">
> hardcoded git handling which<br class="">
> > makes it look problematic from many angles (picking a proper git,<br class="">
> > selecting exact way of storing information, creating repository,<br class="">
> replacing the file etc etc).<br class="">
> ><br class="">
> > Just dumping information in a way that allows easy subsequent<br class="">
> machine processing<br class="">
> > seems to be a more flexible, less cluttered and overall clean<br class="">
> solution that allows to avoid<br class="">
> > making any of "user interface" decisions mentioned above.<br class="">
> ><br class="">
> > We need to understand why git-commit-after-all works faster than<br class="">
> print-after-all.<br class="">
> Made an interesting experiment today and extended your<br class="">
> git-commit-after-all to avoid issuing<br class="">
> any git commands if git-repo starts with "/dev/".<br class="">
><br class="">
> With git-repo=/dev/stderr it becomes functionally equivalent to<br class="">
> print-after-all+print-module-scope,<br class="">
> dumping module into stderr after each pass.<br class="">
><br class="">
> On my testcase:<br class="">
><br class="">
> # first normal git-commit-after-all execution<br class="">
> ] rm -rf test-git; time $RR/bin/opt -O1 some-ir.ll -disable-output<br class="">
> -git-commit-after-all -git-repo=./test-git<br class="">
><br class="">
> real 0m7.172s<br class="">
> user 0m6.303s<br class="">
> sys 0m0.902s<br class="">
> # then "printing" git-commit-after-all execution<br class="">
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output<br class="">
> -git-commit-after-all -git-repo=/dev/stderr 2>&1 | grep -c '^; ModuleID'<br class="">
> 615<br class="">
><br class="">
> real 0m2.893s<br class="">
> user 0m2.859s<br class="">
> sys 0m0.356s<br class="">
> # and finally print-after-all<br class="">
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output -print-after-all<br class="">
> -print-module-scope 2>&1 | grep -c "^; ModuleID"<br class="">
> 526<br class="">
><br class="">
> real 2m8.024s<br class="">
> user 0m55.933s<br class="">
> sys 3m19.253s<br class="">
> ]<br class="">
> Ugh... 60x???<br class="">
> Now, I'm set to analyze this astonishing difference that threatens my<br class="">
> sanity (while I'm still sane ... hopefully).<br class="">
><br class="">
> regards,<br class="">
> Fedor.<br class="">
> PS btw, I checked /dev/null - and it works faster than /dev/stderr as<br class="">
> expected :)<br class="">
><br class="">
> > I dont believe in magic... yet :)<br class="">
> ><br class="">
> > And, btw, thanks for both the idea and the patch.<br class="">
> ><br class="">
> > regards,<br class="">
> > Fedor.<br class="">
> ><br class="">
> > On 03/16/2018 12:03 AM, Alexandre Isoard wrote:<br class="">
> >> If this is faster than -print-after-all we may actually consider<br class="">
> pushing that in the code base then? (after diligent code review of<br class="">
> course)<br class="">
> >><br class="">
> >> Note that it uses the same printing method as -print-after-all:<br class="">
> >> - create a pass of the same pass kind as the pass we just ran<br class="">
> >> - use Module::print(raw_ostream) to print (except -print-after-all<br class="">
> only print the concerned part and into stdout)<br class="">
> >><br class="">
> >> If there is improvement to be done to print-after-all it might also<br class="">
> improve git-commit-after-all. (unless that only improve speed when<br class="">
> printing constructs smaller than module)<br class="">
> >><br class="">
> >> In any case, it is, to me, much more usable (and extensible) than<br class="">
> -print-after-all. But requires git to be in PATH (I'm curious if that<br class="">
> works on Windows).<br class="">
> >><br class="">
> >> On Thu, Mar 15, 2018 at 1:35 PM, Daniel Sanders<br class="">
> <<a href="mailto:daniel_l_sanders@apple.com" target="_blank" class="">daniel_l_sanders@apple.com</a>> wrote:<br class="">
> >><br class="">
> >> Does <a href="https://reviews.llvm.org/D44132" rel="noreferrer" target="_blank" class="">https://reviews.llvm.org/D44132</a> help at all?<br class="">
> >><br class="">
> >><br class="">
> >>> On 15 Mar 2018, at 09:16, Philip Reames via llvm-dev<br class="">
> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class="">
> >>><br class="">
> >>> The most likely answer is that the printer used by<br class="">
> print-after-all is slow. I know there were some changes made around<br class="">
> passing in some form of state cache (metadata related?) and that<br class="">
> running printers without doing so work, but are dog slow. I suspect<br class="">
> the print-after-all support was never updated. Look at what we do for<br class="">
> the normal IR emission "-S" and see if print-after-all is out of sync.<br class="">
> >>><br class="">
> >>> Philip<br class="">
> >>><br class="">
> >>> On 03/15/2018 08:45 AM, Alexandre Isoard via llvm-dev wrote:<br class="">
> >>>> Huh. Great! 😁<br class="">
> >>>><br class="">
> >>>> I don't believe my poor excuse from earlier (else we should<br class="">
> map all pipes into files!), but I'm curious why we spend less time in<br class="">
> system mode when going through file than pipe. Maybe /dev/null is not<br class="">
> as efficient as we might think? I can't believe I'm saying that...<br class="">
> >>>><br class="">
> >>>> On Thu, Mar 15, 2018, 08:25 Fedor Sergeev<br class="">
> <<a href="mailto:fedor.sergeev@azul.com" target="_blank" class="">fedor.sergeev@azul.com</a>> wrote:<br class="">
> >>>><br class="">
> >>>> Well, git by itself is so focused on performance, so its<br class="">
> not surprising<br class="">
> >>>> to me that even using git add/git commit does not cause<br class="">
> >>>> performance penalties.<br class="">
> >>>><br class="">
> >>>><br class="">
> >>>> Sure, but still, I write more stuff (entire module) into a<br class="">
> slower destination (file). Even ignoring git execution time it's<br class="">
> counter intuitive.<br class="">
> >>>><br class="">
> >>>> The only difference is that while I write more, it overwrite<br class="">
> itself continuously, instead of being a long linear steam. I was<br class="">
> thinking of mmap the file instead of going through our raw_stream, but<br class="">
> maybe that's unnecessary then...<br class="">
> >>>><br class="">
> >>>><br class="">
> >>>><br class="">
> >>>> _______________________________________________<br class="">
> >>>> LLVM Developers mailing list<br class="">
> >>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
> >>>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
> >>><br class="">
> >>> _______________________________________________<br class="">
> >>> LLVM Developers mailing list<br class="">
> >>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
> >>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
> >><br class="">
> >><br class="">
> >><br class="">
> >><br class="">
> >> --<br class="">
> >> Alexandre Isoard<br class="">
> ><br class="">
> ><br class="">
> ><br class="">
> > _______________________________________________<br class="">
> > LLVM Developers mailing list<br class="">
> > <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
> > <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
><br class="">
><br class="">
> _______________________________________________<br class="">
> LLVM Developers mailing list<br class="">
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</blockquote></div>
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><b class="">Alexandre Isoard</b><br class=""></div></div>
_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></div></blockquote></div><br class=""></div></div></div></body></html>