[llvm-dev] Commit module to Git after each Pass
Fedor Sergeev via llvm-dev
llvm-dev at lists.llvm.org
Wed Mar 21 07:15:18 PDT 2018
On 03/21/2018 03:23 PM, Jeremy Lakeman wrote:
> Do you really need to write the entire module to a single file? (Hence
> my earlier hint...)
First, prime motivation for this particular experiment was to make
Alexandre's implementation
match -print-after-all functionality, which dumps entire set of modules
into a single file.
Then, even for my own initial implementation of "ala git-commit"
functionality I would
just dump everything into a stream and postprocess it (perhaps making it
a bit easier to postprocess
by properly marking the stream, but still a single stream).
> Why not write out a separate file for each def, so you don't need to
> dump functions that haven't changed?
Because I wouldnt want to figure out what has or what has not changed.
I would rather leave it to git itself :)
And frankly, the problem I'm seeing right now is not related to details
of a particular dump organization.
I'm seeing a huge performance difference in what I consider to be very
similar implementations
of module IR printing...
regards,
Fedor.
>
> On Wed, Mar 21, 2018 at 8:38 PM, Fedor Sergeev via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
> On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote:
> > git-commit-after-all solution has one serious issue - it has a
> hardcoded git handling which
> > makes it look problematic from many angles (picking a proper git,
> > selecting exact way of storing information, creating repository,
> replacing the file etc etc).
> >
> > Just dumping information in a way that allows easy subsequent
> machine processing
> > seems to be a more flexible, less cluttered and overall clean
> solution that allows to avoid
> > making any of "user interface" decisions mentioned above.
> >
> > We need to understand why git-commit-after-all works faster than
> print-after-all.
> Made an interesting experiment today and extended your
> git-commit-after-all to avoid issuing
> any git commands if git-repo starts with "/dev/".
>
> With git-repo=/dev/stderr it becomes functionally equivalent to
> print-after-all+print-module-scope,
> dumping module into stderr after each pass.
>
> On my testcase:
>
> # first normal git-commit-after-all execution
> ] rm -rf test-git; time $RR/bin/opt -O1 some-ir.ll -disable-output
> -git-commit-after-all -git-repo=./test-git
>
> real 0m7.172s
> user 0m6.303s
> sys 0m0.902s
> # then "printing" git-commit-after-all execution
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output
> -git-commit-after-all -git-repo=/dev/stderr 2>&1 | grep -c '^;
> ModuleID'
> 615
>
> real 0m2.893s
> user 0m2.859s
> sys 0m0.356s
> # and finally print-after-all
> ] time $RR/bin/opt -O1 some-ir.ll -disable-output -print-after-all
> -print-module-scope 2>&1 | grep -c "^; ModuleID"
> 526
>
> real 2m8.024s
> user 0m55.933s
> sys 3m19.253s
> ]
> Ugh... 60x???
> Now, I'm set to analyze this astonishing difference that threatens
> my sanity (while I'm still sane ... hopefully).
>
> regards,
> Fedor.
> PS btw, I checked /dev/null - and it works faster than /dev/stderr
> as expected :)
>
>
> > I dont believe in magic... yet :)
> >
> > And, btw, thanks for both the idea and the patch.
> >
> > regards,
> > Fedor.
> >
> > On 03/16/2018 12:03 AM, Alexandre Isoard wrote:
> >> If this is faster than -print-after-all we may actually
> consider pushing that in the code base then? (after diligent code
> review of course)
> >>
> >> Note that it uses the same printing method as -print-after-all:
> >> - create a pass of the same pass kind as the pass we just ran
> >> - use Module::print(raw_ostream) to print (except
> -print-after-all only print the concerned part and into stdout)
> >>
> >> If there is improvement to be done to print-after-all it might
> also improve git-commit-after-all. (unless that only improve speed
> when printing constructs smaller than module)
> >>
> >> In any case, it is, to me, much more usable (and extensible)
> than -print-after-all. But requires git to be in PATH (I'm curious
> if that works on Windows).
> >>
> >> On Thu, Mar 15, 2018 at 1:35 PM, Daniel Sanders
> <daniel_l_sanders at apple.com <mailto:daniel_l_sanders at apple.com>>
> wrote:
> >>
> >> Does https://reviews.llvm.org/D44132
> <https://reviews.llvm.org/D44132> help at all?
> >>
> >>
> >>> On 15 Mar 2018, at 09:16, Philip Reames via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >>>
> >>> The most likely answer is that the printer used by
> print-after-all is slow. I know there were some changes made
> around passing in some form of state cache (metadata related?) and
> that running printers without doing so work, but are dog slow. I
> suspect the print-after-all support was never updated. Look at
> what we do for the normal IR emission "-S" and see if
> print-after-all is out of sync.
> >>>
> >>> Philip
> >>>
> >>> On 03/15/2018 08:45 AM, Alexandre Isoard via llvm-dev wrote:
> >>>> Huh. Great! 😁
> >>>>
> >>>> I don't believe my poor excuse from earlier (else we
> should map all pipes into files!), but I'm curious why we spend
> less time in system mode when going through file than pipe. Maybe
> /dev/null is not as efficient as we might think? I can't believe
> I'm saying that...
> >>>>
> >>>> On Thu, Mar 15, 2018, 08:25 Fedor Sergeev
> <fedor.sergeev at azul.com <mailto:fedor.sergeev at azul.com>> wrote:
> >>>>
> >>>> Well, git by itself is so focused on performance, so
> its not surprising
> >>>> to me that even using git add/git commit does not cause
> >>>> performance penalties.
> >>>>
> >>>>
> >>>> Sure, but still, I write more stuff (entire module) into
> a slower destination (file). Even ignoring git execution time it's
> counter intuitive.
> >>>>
> >>>> The only difference is that while I write more, it
> overwrite itself continuously, instead of being a long linear
> steam. I was thinking of mmap the file instead of going through
> our raw_stream, but maybe that's unnecessary then...
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> LLVM Developers mailing list
> >>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >>>
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >>
> >>
> >>
> >>
> >> --
> >> Alexandre Isoard
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180321/9debf1f7/attachment.html>
More information about the llvm-dev
mailing list