<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0"></p>
<div>> FWIW: We could also just have a mode that dumps 1 file per pass. That is enough to make it convenient/easy to run diff between passes. (And if you wanted to you could still</div>
<div>> make a git repository out of it with an external script).<br class="">
<div class="">><br class="">
</div>
> - Matthias</div>
<br>
<p></p>
<p style="margin-top:0;margin-bottom:0">I have done this before and would strongly encourage this approach as opposed to direct emission to std[out|err] or directly involving a source control system. The most convenient way was to add an additional option,
-print-to-files, which modified the behavior of -print-after-all, -print-before-all, etc. The filename was constructed by massaging the pass name to comply with file system naming conventions and prepending a monotonically increasing integer (with suitable
leading zeros) plus "bef" or "aft" to indicate sequencing. The only awkward part was modifying createPrinterPass to accept a filename, which had to be done because otherwise you end up having to keep each stream open from the time you setup the pass pipeline
until the printing pass actually runs.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">-Troy<br>
</p>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> llvm-dev <llvm-dev-bounces@lists.llvm.org> on behalf of mbraun via llvm-dev <llvm-dev@lists.llvm.org><br>
<b>Sent:</b> Thursday, June 14, 2018 3:48 PM<br>
<b>To:</b> Alexandre Isoard<br>
<b>Cc:</b> llvm-dev<br>
<b>Subject:</b> Re: [llvm-dev] Commit module to Git after each Pass</font>
<div> </div>
</div>
<meta content="text/html; charset=utf-8">
<div class="" style="word-wrap:break-word; line-break:after-white-space">FWIW: We could also just have a mode that dumps 1 file per pass. That is enough to make it convenient/easy to run diff between passes.
<div class="">(And if you wanted to you could still make a git repository out of it with an external script).<br class="">
<div class=""><br class="">
</div>
<div class="">- Matthias<br class="">
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 14, 2018, at 10:49 AM, Alexandre Isoard via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="OWAAutoLink" id="LPlnk278353" previewremoved="true">llvm-dev@lists.llvm.org</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hello,
<div class=""><br class="">
</div>
<div class="">Just an update on that. I am personally using -git-commit-after-all *as-is* extremely frequently (combined with "git filter-branch" and "opt -S -instnamer" it is extremely useful).</div>
<div class="">I unfortunately won't have time to write a better implementation of that, and I agree "git fast-import" seems the way to go. If anybody is motivated enough to do so, feel free.</div>
<div class=""><br class="">
</div>
<div class="">Best regard!</div>
</div>
<br class="">
<div class="x_gmail_quote">
<div dir="ltr" class="">On Thu, Mar 22, 2018 at 10:38 AM Reid Kleckner via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="OWAAutoLink" id="LPlnk392232" previewremoved="true">llvm-dev@lists.llvm.org</a>> wrote:<br class="">
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr" class="">Obviously, we do not want all stderr output to be buffered. However, I think it would be great to change Function::print and Module::print to call raw_ostream::SetBuffered / raw_ostream::SetUnbuffered before and after printing. I guess
if the original stream was buffered we don't want to mark it unbuffered, so we may need to tweak the raw_ostream interface. Looks easy, though.</div>
<br class="">
<br class="">
<div class="x_gmail_quote">
<div dir="ltr" class="">On Thu, Mar 22, 2018 at 8:06 AM Fedor Sergeev via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk29324" previewremoved="true">llvm-dev@lists.llvm.org</a>> wrote:<br class="">
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Oh, well... as usually the answer appears to be pretty obvious.<br class="">
99% of the time is spent inside the plain write.<br class="">
<br class="">
-print-after-all prints into llvm::errs(), which is an *unbuffered*<br class="">
raw_fd_stream.<br class="">
And -git-commit-after-all opens a *buffered* raw_fd_stream.<br class="">
<br class="">
As soon as I hacked -print-after-all to use a buffered stream to stderr<br class="">
performance went<br class="">
up to the normal expected values:<br class="">
<br class="">
] time bin/opt -O1 big-ir.ll -disable-output -print-after-all<br class="">
-print-module-scope 2>&1 | grep -c "^; ModuleID"<br class="">
526<br class="">
<br class="">
real 0m2.363s<br class="">
user 0m2.373s<br class="">
sys 0m0.271s<br class="">
]<br class="">
<br class="">
So, the morale of this story is - we should not be printing module IR<br class="">
into dbgs/errs().<br class="">
<br class="">
And then the idea of streaming IR module dumps into a buffered stream<br class="">
and then postprocessing seems<br class="">
to be a right one.<br class="">
<br class="">
regards,<br class="">
Fedor.<br class="">
<br class="">
On 03/21/2018 01:08 PM, Fedor Sergeev via llvm-dev wrote:<br class="">
> On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote:<br class="">
> > git-commit-after-all solution has one serious issue - it has a<br class="">
> hardcoded git handling which<br class="">
> > makes it look problematic from many angles (picking a proper git,<br class="">
> > selecting exact way of storing information, creating repository,<br class="">
> replacing the file etc etc).<br class="">
> ><br class="">
> > Just dumping information in a way that allows easy subsequent<br class="">
> machine processing<br class="">
> > seems to be a more flexible, less cluttered and overall clean<br class="">
> solution that allows to avoid<br class="">
> > making any of "user interface" decisions mentioned above.<br class="">
> ><br class="">
> > We need to understand why git-commit-after-all works faster than<br class="">
> print-after-all.<br class="">
> Made an interesting experiment today and extended your<br class="">
> git-commit-after-all to avoid issuing<br class="">
> any git commands if git-repo starts with "/dev/".<br class="">
><br class="">
> With git-repo=/dev/stderr it becomes functionally equivalent to<br class="">
> print-after-all+print-module-scope,<br class="">
> dumping module into stderr after each pass.<br class="">
><br class="">
> On my testcase:<br class="">
><br class="">
> # first normal git-commit-after-all execution<br class="">
> ] rm -rf test-git; time $RR/bin/opt -O1 some-ir.ll -disable-output<br class="">
> -git-commit-after-all -git-repo=./test-git<br class="">
><br class="">
> real 0m7.172s<br class="">
> user 0m6.303s<br class="">
> sys 0m0.902s<br class="">
> # then "printing" git-commit-after-all execution<br class="">
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output<br class="">
> -git-commit-after-all -git-repo=/dev/stderr 2>&1 | grep -c '^; ModuleID'<br class="">
> 615<br class="">
><br class="">
> real 0m2.893s<br class="">
> user 0m2.859s<br class="">
> sys 0m0.356s<br class="">
> # and finally print-after-all<br class="">
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output -print-after-all<br class="">
> -print-module-scope 2>&1 | grep -c "^; ModuleID"<br class="">
> 526<br class="">
><br class="">
> real 2m8.024s<br class="">
> user 0m55.933s<br class="">
> sys 3m19.253s<br class="">
> ]<br class="">
> Ugh... 60x???<br class="">
> Now, I'm set to analyze this astonishing difference that threatens my<br class="">
> sanity (while I'm still sane ... hopefully).<br class="">
><br class="">
> regards,<br class="">
> Fedor.<br class="">
> PS btw, I checked /dev/null - and it works faster than /dev/stderr as<br class="">
> expected :)<br class="">
><br class="">
> > I dont believe in magic... yet :)<br class="">
> ><br class="">
> > And, btw, thanks for both the idea and the patch.<br class="">
> ><br class="">
> > regards,<br class="">
> > Fedor.<br class="">
> ><br class="">
> > On 03/16/2018 12:03 AM, Alexandre Isoard wrote:<br class="">
> >> If this is faster than -print-after-all we may actually consider<br class="">
> pushing that in the code base then? (after diligent code review of<br class="">
> course)<br class="">
> >><br class="">
> >> Note that it uses the same printing method as -print-after-all:<br class="">
> >> - create a pass of the same pass kind as the pass we just ran<br class="">
> >> - use Module::print(raw_ostream) to print (except -print-after-all<br class="">
> only print the concerned part and into stdout)<br class="">
> >><br class="">
> >> If there is improvement to be done to print-after-all it might also<br class="">
> improve git-commit-after-all. (unless that only improve speed when<br class="">
> printing constructs smaller than module)<br class="">
> >><br class="">
> >> In any case, it is, to me, much more usable (and extensible) than<br class="">
> -print-after-all. But requires git to be in PATH (I'm curious if that<br class="">
> works on Windows).<br class="">
> >><br class="">
> >> On Thu, Mar 15, 2018 at 1:35 PM, Daniel Sanders<br class="">
> <<a href="mailto:daniel_l_sanders@apple.com" target="_blank" class="OWAAutoLink" id="LPlnk652580" previewremoved="true">daniel_l_sanders@apple.com</a>> wrote:<br class="">
> >><br class="">
> >> Does <a href="https://reviews.llvm.org/D44132" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk758822" previewremoved="true">
https://reviews.llvm.org/D44132</a> help at all?<br class="">
> >><br class="">
> >><br class="">
> >>> On 15 Mar 2018, at 09:16, Philip Reames via llvm-dev<br class="">
> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk436836" previewremoved="true">llvm-dev@lists.llvm.org</a>> wrote:<br class="">
> >>><br class="">
> >>> The most likely answer is that the printer used by<br class="">
> print-after-all is slow. I know there were some changes made around<br class="">
> passing in some form of state cache (metadata related?) and that<br class="">
> running printers without doing so work, but are dog slow. I suspect<br class="">
> the print-after-all support was never updated. Look at what we do for<br class="">
> the normal IR emission "-S" and see if print-after-all is out of sync.<br class="">
> >>><br class="">
> >>> Philip<br class="">
> >>><br class="">
> >>> On 03/15/2018 08:45 AM, Alexandre Isoard via llvm-dev wrote:<br class="">
> >>>> Huh. Great! 😁<br class="">
> >>>><br class="">
> >>>> I don't believe my poor excuse from earlier (else we should<br class="">
> map all pipes into files!), but I'm curious why we spend less time in<br class="">
> system mode when going through file than pipe. Maybe /dev/null is not<br class="">
> as efficient as we might think? I can't believe I'm saying that...<br class="">
> >>>><br class="">
> >>>> On Thu, Mar 15, 2018, 08:25 Fedor Sergeev<br class="">
> <<a href="mailto:fedor.sergeev@azul.com" target="_blank" class="OWAAutoLink" id="LPlnk824024" previewremoved="true">fedor.sergeev@azul.com</a>> wrote:<br class="">
> >>>><br class="">
> >>>> Well, git by itself is so focused on performance, so its<br class="">
> not surprising<br class="">
> >>>> to me that even using git add/git commit does not cause<br class="">
> >>>> performance penalties.<br class="">
> >>>><br class="">
> >>>><br class="">
> >>>> Sure, but still, I write more stuff (entire module) into a<br class="">
> slower destination (file). Even ignoring git execution time it's<br class="">
> counter intuitive.<br class="">
> >>>><br class="">
> >>>> The only difference is that while I write more, it overwrite<br class="">
> itself continuously, instead of being a long linear steam. I was<br class="">
> thinking of mmap the file instead of going through our raw_stream, but<br class="">
> maybe that's unnecessary then...<br class="">
> >>>><br class="">
> >>>><br class="">
> >>>><br class="">
> >>>> _______________________________________________<br class="">
> >>>> LLVM Developers mailing list<br class="">
> >>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk562759" previewremoved="true">
llvm-dev@lists.llvm.org</a><br class="">
> >>>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk913759" previewremoved="true">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
> >>><br class="">
> >>> _______________________________________________<br class="">
> >>> LLVM Developers mailing list<br class="">
> >>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk378074" previewremoved="true">
llvm-dev@lists.llvm.org</a><br class="">
> >>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk423852" previewremoved="true">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
> >><br class="">
> >><br class="">
> >><br class="">
> >><br class="">
> >> --<br class="">
> >> Alexandre Isoard<br class="">
> ><br class="">
> ><br class="">
> ><br class="">
> > _______________________________________________<br class="">
> > LLVM Developers mailing list<br class="">
> > <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk827401" previewremoved="true">
llvm-dev@lists.llvm.org</a><br class="">
> > <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk749000" previewremoved="true">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
><br class="">
><br class="">
> _______________________________________________<br class="">
> LLVM Developers mailing list<br class="">
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk576280" previewremoved="true">
llvm-dev@lists.llvm.org</a><br class="">
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk736391" previewremoved="true">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk461700" previewremoved="true">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk947562" previewremoved="true">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</blockquote>
</div>
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="OWAAutoLink" id="LPlnk796307" previewremoved="true">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="OWAAutoLink" id="LPlnk952731" previewremoved="true">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr" class=""><b class="">Alexandre Isoard</b><br class="">
</div>
</div>
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="OWAAutoLink" id="LPlnk334952" previewremoved="true">llvm-dev@lists.llvm.org</a><br class="">
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>