[cfe-dev] Why clang needs to fork into itself?

[Alp Toker]

On 29/01/2014 20:40, Richard Smith wrote:
> On Wed Jan 29 2014 at 11:37:22 AM, Alp Toker <alp at nuanti.com 
> <mailto:alp at nuanti.com>> wrote:
>
>
>     On 29/01/2014 18:15, Ted Kremenek wrote:
>     > On Jan 29, 2014, at 9:55 AM, Richard Smith <metafoo at gmail.com
>     <mailto:metafoo at gmail.com>
>     > <mailto:metafoo at gmail.com <mailto:metafoo at gmail.com>>> wrote:
>     >
>     >> On Wed Jan 29 2014 at 12:35:23 AM, Ted Kremenek
>     <kremenek at apple.com <mailto:kremenek at apple.com>
>     >> <mailto:kremenek at apple.com <mailto:kremenek at apple.com>>> wrote:
>     >>
>     >> On Jan 28, 2014, at 11:58 PM, Richard Smith
>     >> <richard at metafoo.co.uk <mailto:richard at metafoo.co.uk>
>     <mailto:richard at metafoo.co.uk <mailto:richard at metafoo.co.uk>>> wrote:
>     >>
>     >>> That said, most users of the compiler don't actually *want* a
>     >>> driver. They want just a .c* -> .o compiler, not a
>     >>> multi-source-file cross-language compiler + assembler + linker +
>     >>> kitchen sink, and it makes sense to me to optimize for the
>     >>> single-source, -c case.
>     >>
>     >> I don’t really know if it matters what most users “want.” We
>     >> still need to support the kitchen sink model; it’s never going to
>     >> go away.
>     >>
>     >>
>     >> I think it does matter, because it helps us set our priorities. In
>     >> particular, if we can make single-source -c builds measurably
>     faster,
>     >> that has benefits for a lot of users, even if we leave all other
>     >> modes alone.
>     >>
>     >> By “optimize”, what specifically do you propose/suggest?
>     >>
>     >>
>     >> Nothing concrete, merely that we shouldn't dismiss this out of
>     hand.
>     >> For instance, if we can tell people to run -c --disable-fork for a
>     >> measurable speedup (with a corresponding degradation in crash
>     >> recovery), that might be valuable to some.
>     >
>     > I agree that making single source builds faster is a case worth
>     > optimizing.
>
>     Hi Richard, Ted,
>
>     I have a very simple solution to get started with:
>
>     + bool TryInProcess =
>     getArgs().hasArg(options::OPT_fno_crash_diagnostics);
>     + if (TryInProcess && C.getExecutable() ==
>     getDriver().getClangProgramPath()) {
>     + const ArgStringList &Arguments = C.getArguments();
>     ...
>     + Res = main(Argv.size() - 1, Argv.data());
>     + } else
>
>     This passes all but one of the tests (which appears to be a
>     genuine bug
>     in the preprocessor that I'm looking to fix now).
>
>     All -cc1, -cc1as, -E and a couple of other invocations get run
>     in-process with no apparent regressions.
>
>     This setup parallels the way we skip expensive checks in Parse and
>     Sema
>     when we know a diagnostic is ignored and it shouldn't have any
>     observable effect to the user other than fewer PIDs and invocations.
>
>     I'd like to move forward with this so we can start to experiment with
>     in-process operation focusing on the single input case Richard
>     mentioned.
>
>     It's not particularly hi-tech, but that's what makes it kind of neat.
>     What do you think?
>
>
> I'd be surprised if it works as-is (I'd expect we have globals that 
> misbehave when treated this way).

I was just as surprised. It passes the tests and selfhost with only the 
main process and the occasional ld invocation. Even clang-cl.exe works. 
The one problem with a closed stdout fd was easily worked around.

My guess is that the tricky bugs have simply been ironed out by Tooling 
which already supports multiple uses in process.


> Also, invoking 'main' in this way is ill-formed, so we'd need to 
> extract a 'clang_recursive_main' or similar (and mark it 
> __attribute__((weak)) so we don't require tools to provide it).

Indeed, that snippet was just a redaction for the email. (Shame we don't 
warn for main() calls though.)

I've actually done it as a function pointer callback map so you can 
override lld too or anything else that you happen to have an entry point 
for which makes it easy to test with an installed set of clang libraries.

>
> This seems like a good way forward for extracting performance numbers, 
> and on the basis of those we can decide if this is worth pursuing. I 
> don't think it's worth upstreaming until you have such numbers (maybe 
> you could mail out a patch so interested parties can profile it?).

Yes, will provide a more concrete patch shortly so others can chip in.

(Overall I'm still a fan of the forking, but this has been pretty 
interesting and the two approaches appear complementary.)

Alp.


>     Alp.
>
>
>     >
>     >> Even the .c -> .o compiler benefits from process isolation for
>     >> the purpose of crash reporting. Perhaps there are alternate
>     >> solutions for reporting crashes, but they are a bit speculative
>     >> at this point.
>     >>
>     >> I also haven’t seen any numbers mentioned on this thread that
>     >> justify “optimizing” anything, at least from a performance
>     >> perspective. I’m not saying there isn’t a possible performance
>     >> win here by eliminating the posix_spawn call, but that hasn’t
>     >> been justified yet (or if it has, I apologize that I haven’t seen
>     >> it). Or perhaps this discussion isn’t about optimizing
>     >> performance, but refining the driver design. In that case, it’s
>     >> not clear to me what specific problems are being tackled here
>     >> other than possibly making the compiler a little easier to debug
>     >> or profile.
>     >>
>     >>
>     >> I think the changes proposed on this thread will make the driver
>     >> design more complex. I'd also like to see some numbers.
>     >
>     > Agreed.
>     >
>     >
>     > _______________________________________________
>     > cfe-dev mailing list
>     > cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
>     > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>     --
>     http://www.nuanti.com
>     the browser experts
>

-- 
http://www.nuanti.com
the browser experts