[cfe-dev] Why clang needs to fork into itself?

[Sean Silva]

On Wed, Jan 29, 2014 at 3:26 PM, Alp Toker <alp at nuanti.com> wrote:

>
> On 29/01/2014 20:06, Sean Silva wrote:
>
>
>>
>>
>> On Tue, Jan 28, 2014 at 12:45 PM, Nico Weber <thakis at chromium.org<mailto:
>> thakis at chromium.org>> wrote:
>>
>>     I believe there was a thread about not forking the driver a while
>>     ago, but I'm unable to find it. As far as I remember, Chris
>>     Lattner wanted to get rid of it for aesthetic reasons and to save
>>     the milliseconds of overhead it adds (I think doing this has
>>     originally been the plan, see the "fork/exec" section on
>>     http://lists.cs.uiuc.edu/pipermail/cfe-dev/2009-December/007211.html
>> ).
>>     At the end, the decision was made to keep the subprocess for cc1,
>>     but I don't remember all the reasons. I think crash reporting was
>>     part of the discussion, but there was half a plan to keep that
>>     feature with in-process crash reporting somehow.
>>
>>     Maybe someone still has a copy of that thread in their inbox?
>>     Keywords "lattner dgregor cc1 crash fork exec" or similar might
>>     find it.
>>
>>
>> The thread you linked to predates my LLVM involvement, but I remember
>> another thread (long time ago) that I was involved in where basically it
>> was asked "is this a measurable slowdown?" (my me or someone else, I
>> forget) and nobody could provide real numbers supporting that it was a
>> measurable slowdown (even on Windows). Can't see to find the thread in my
>> email though.
>>
>> Do we have real performance numbers demonstrating a slowdown at this
>> point?
>>
>
> Hi Sean,
>
> I have a patch now that pushes all self-invocations in-process without
> doing anything else clever such as reusing FileManager stat caches or
> in-memory object files (6 line diff, posted in another part of this thread).
>
> Haven't had a chance to time it yet, but it doesn't look like an
> earth-shattering win on OS X (5 reruns of make check show around 2 seconds
> saved from a 30 second run as a very early indication).
>

However, make check runs the tests too (with the just-built binary, rather
than the one under test, which is confusing). It would be better to isolate
just the build time with the modified compiler (and ensuring that both the
control compiler and the modified compiler are tested building the same
source tree).

The fact that `make check` also runs the test suites would make the speedup
even more impressive. 2 sec out of 30 sec corresponds to ~6% improvement.
Estimating that 50% of the time of `make check` is spent running the test
suite (which takes the same amount of time either way, since the just-built
compiler is identical), this is a >10% improvement in compilation speed. If
the test suite takes 90% of the time, then it would correspond to a >60%
improvement in compilation speed (unlikely).

When you do more thorough testing, could you please provide the raw data so
that geeks like me can analyze it? I've grown very cautious about believing
in people's statistical results without seeing the data and double-checking
(at least doing a distribution plot or such).

-- Sean Silva


>
> On the other hand it's been fun to debug the driver with this setup and I
> suspect there'll be a bigger win on Windows.
>
> Another patch I'm cooking up just to see "what happens", is to use the
> IPCChannel facility recently added to LLVM to avoid the driver invocation
> itself.
>
> Alp.
>
>
>
>> -- Sean Silva
>>
>>
>>
>>     On Tue, Jan 28, 2014 at 7:19 AM, Rafael Espíndola
>>     <rafael.espindola at gmail.com <mailto:rafael.espindola at gmail.com>>
>>
>>     wrote:
>>
>>         On 27 January 2014 17:27, Reid Kleckner <rnk at google.com
>>         <mailto:rnk at google.com>> wrote:
>>         > As an alternative, on Windows we could rig up some kind of
>>         SEH filter to do
>>         > crash recovery.  Then we could save the subprocess
>>         invocation and speed
>>         > things up.
>>
>>         Breakpad is used by both chrome and firefox for this. If going
>>         this
>>         path, please make sure the same technique is used for all systems.
>>
>>         Cheers,
>>         Rafale
>>         _______________________________________________
>>         cfe-dev mailing list
>>         cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
>>
>>         http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>>
>>     _______________________________________________
>>     cfe-dev mailing list
>>     cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
>>
>>     http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>
> --
> http://www.nuanti.com
> the browser experts
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140129/3d8222ff/attachment.html>