r261774 - Bail on compilation as soon as a job fails.
David Blaikie via cfe-commits
cfe-commits at lists.llvm.org
Mon Feb 29 12:50:12 PST 2016
On Mon, Feb 29, 2016 at 11:19 AM, Justin Lebar via cfe-commits <
cfe-commits at lists.llvm.org> wrote:
> > Can you expand on this? The idea is that those implicit deps are in
> addition to regular dependencies, not a replacement.
>
> Sure.
>
> What I was trying to say is, suppose you have two CUDA device
> compilations. You want device compilation B to bail if device
> compilation A fails. So what are the implicit dependencies of B?
>
> Well, B must implicitly depend on everything in A that may trigger a
> failure. Which is, I recall, four actions: preprocessor, compilation,
> backend, and assemble. It may in fact be one or two more because we
> have these CUDADeviceActions that wrap other actions, and maybe we'd
> have to depend on those too.
>
Yeah, that doesn't sound quite right - presumably our link action doesn't
depend on all those steps, just the last one. If any others fail, the rest
can't run (because they depend on them in turn) so the link won't run.
(but, yeah, I'm no Driver expert by any means..)
>
> Or maybe the thing you're proposing is smart enough that we only need
> to depend on the assemble action and it will infer the rest correctly.
> Either way, I was just saying that if you add another action, C, it
> now has to depend on everything from A and B, and so on.
>
> > One problem I have with this change is that it breaks part of my patch
> ;-)
>
> Now it's my turn to admit that I didn't quite follow the rest of this. :)
>
> > if you're in the office tomorrow, maybe we could discuss this for half
> an hour or so in person?
>
> Yes, absolutely, let's talk irl and figure this out.
>
> On Mon, Feb 29, 2016 at 10:43 AM, Nico Weber <thakis at chromium.org> wrote:
> > On Sun, Feb 28, 2016 at 3:40 PM, Justin Lebar <jlebar at google.com> wrote:
> >>
> >> On Sun, Feb 28, 2016 at 1:46 PM, Nico Weber <thakis at chromium.org>
> wrote:
> >> > Do you think something like the implicit inputs thing in
> >> > http://reviews.llvm.org/D17695 could work for you as well, instead of
> >> > this
> >> > patch?
> >>
> >> Having read just the patch description, I think it would be workable,
> >> although it might not be particularly clean. I think you'd have to
> >> make most (all?) of the intermediate actions associated with a
> >> compilation be implicit inputs to all later actions in all other
> >> compilations. Otherwise things like saving vs. not saving temps would
> >> give you different behavior.
> >
> >
> > Can you expand on this? The idea is that those implicit deps are in
> addition
> > to regular dependencies, not a replacement.
> >
> >>
> >> It seems like kind of a fragile way to get this behavior.
> >>
> >> > Then we don't have to forever promise to compile all .cc input files
> >> > serially.
> >>
> >> I've thought about this some, since we don't really want this even for
> >> CUDA. With CUDA, you commonly want to compile the device code for
> >> different architectures, and it would be reasonable to do those
> >> compilations in parallel.
> >>
> >> What I think may be a reasonable promise, at least for CUDA, is that
> >> we will behave *as if* we're compiling in series. Which just means
> >> not interleaving diagnostics from different compilations, and not
> >> showing diagnostics from other subcompilations after we hit an error.
> >>
> >> I think basically nobody wants randomly-interleaved diagnostics.
> >> Whether or there's a good use-case where we continue after we hit an
> >> error is a harder question, I'm not sure. But in any case, stopping
> >> after an error shouldn't mean we're forced to serialize, I think.
> >
> >
> > One problem I have with this change is that it breaks part of my patch
> ;-)
> > I'd like to run two commands, one to build a pch, and another to compile
> a
> > source file using the just-built pch, and if the first fails I don't
> want to
> > run the second: `make_pch && compile`. This part still works after your
> > patch. But we also have a flag /fallback that says "if compilation falls,
> > try again with this other compiler. Ideally I want `(make_pch &&
> compile) ||
> > fallback_compile`, but since it's a bit of a corner case and arguably
> good
> > enough, my patch did `make_pch ; compile || fallback`. Your change makes
> it
> > impossible to run commands after one another if one fails, so now this is
> > `make_pch && (compile || fallback)`, i.e. if compilation of the pch fails
> > the fallback compiler won't be invoked.
> >
> > I can try looking at maybe making FallbackCommand a FallbackAction
> instead
> > or something. But since CUDA seems to not fit the internal model of the
> > driver super well (lots of isCUDA() calls in many places), maybe it'd
> make
> > sense to discuss what your requirements are and if it's possible to
> extend
> > the Action / Job / Command abstractions in a way that support CUDA
> without
> > as many special cases. I happen to be in Mountain View – if you're in the
> > office tomorrow, maybe we could discuss this for half an hour or so in
> > person?
> >
> >>
> >>
> >> -Justin
> >>
> >> > On Wed, Feb 24, 2016 at 4:49 PM, Justin Lebar via cfe-commits
> >> > <cfe-commits at lists.llvm.org> wrote:
> >> >>
> >> >> Author: jlebar
> >> >> Date: Wed Feb 24 15:49:28 2016
> >> >> New Revision: 261774
> >> >>
> >> >> URL: http://llvm.org/viewvc/llvm-project?rev=261774&view=rev
> >> >> Log:
> >> >> Bail on compilation as soon as a job fails.
> >> >>
> >> >> Summary:
> >> >> (Re-land of r260448, which was reverted in r260522 due to a test
> >> >> failure
> >> >> in Driver/output-file-cleanup.c that only showed up in fresh builds.)
> >> >>
> >> >> Previously we attempted to be smart; if one job failed, we'd run all
> >> >> jobs that didn't depend on the failing job.
> >> >>
> >> >> Problem is, this doesn't work well for e.g. CUDA compilation without
> >> >> -save-temps. In this case, the device-side and host-side Assemble
> >> >> actions (which actually are responsible for preprocess, compile,
> >> >> backend, and assemble, since we're not saving temps) are necessarily
> >> >> distinct. So our clever heuristic doesn't help us, and we repeat
> every
> >> >> error message once for host and once for each device arch.
> >> >>
> >> >> The main effect of this change, other than fixing CUDA, is that if
> you
> >> >> pass multiple cc files to one instance of clang and you get a compile
> >> >> error, we'll stop when the first cc1 job fails.
> >> >>
> >> >> Reviewers: echristo
> >> >>
> >> >> Subscribers: cfe-commits, jhen, echristo, tra, rafael
> >> >>
> >> >> Differential Revision: http://reviews.llvm.org/D17217
> >> >>
> >> >> Modified:
> >> >> cfe/trunk/lib/Driver/Compilation.cpp
> >> >> cfe/trunk/test/Driver/output-file-cleanup.c
> >> >>
> >> >> Modified: cfe/trunk/lib/Driver/Compilation.cpp
> >> >> URL:
> >> >>
> >> >>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/Compilation.cpp?rev=261774&r1=261773&r2=261774&view=diff
> >> >>
> >> >>
> >> >>
> ==============================================================================
> >> >> --- cfe/trunk/lib/Driver/Compilation.cpp (original)
> >> >> +++ cfe/trunk/lib/Driver/Compilation.cpp Wed Feb 24 15:49:28 2016
> >> >> @@ -163,39 +163,17 @@ int Compilation::ExecuteCommand(const Co
> >> >> return ExecutionFailed ? 1 : Res;
> >> >> }
> >> >>
> >> >> -typedef SmallVectorImpl< std::pair<int, const Command *> >
> >> >> FailingCommandList;
> >> >> -
> >> >> -static bool ActionFailed(const Action *A,
> >> >> - const FailingCommandList &FailingCommands)
> {
> >> >> -
> >> >> - if (FailingCommands.empty())
> >> >> - return false;
> >> >> -
> >> >> - for (FailingCommandList::const_iterator CI =
> >> >> FailingCommands.begin(),
> >> >> - CE = FailingCommands.end(); CI != CE; ++CI)
> >> >> - if (A == &(CI->second->getSource()))
> >> >> - return true;
> >> >> -
> >> >> - for (const Action *AI : A->inputs())
> >> >> - if (ActionFailed(AI, FailingCommands))
> >> >> - return true;
> >> >> -
> >> >> - return false;
> >> >> -}
> >> >> -
> >> >> -static bool InputsOk(const Command &C,
> >> >> - const FailingCommandList &FailingCommands) {
> >> >> - return !ActionFailed(&C.getSource(), FailingCommands);
> >> >> -}
> >> >> -
> >> >> -void Compilation::ExecuteJobs(const JobList &Jobs,
> >> >> - FailingCommandList &FailingCommands)
> >> >> const
> >> >> {
> >> >> +void Compilation::ExecuteJobs(
> >> >> + const JobList &Jobs,
> >> >> + SmallVectorImpl<std::pair<int, const Command *>>
> &FailingCommands)
> >> >> const {
> >> >> for (const auto &Job : Jobs) {
> >> >> - if (!InputsOk(Job, FailingCommands))
> >> >> - continue;
> >> >> const Command *FailingCommand = nullptr;
> >> >> - if (int Res = ExecuteCommand(Job, FailingCommand))
> >> >> + if (int Res = ExecuteCommand(Job, FailingCommand)) {
> >> >> FailingCommands.push_back(std::make_pair(Res,
> FailingCommand));
> >> >> + // Bail as soon as one command fails, so we don't output
> >> >> duplicate
> >> >> error
> >> >> + // messages if we die on e.g. the same file.
> >> >> + return;
> >> >> + }
> >> >> }
> >> >> }
> >> >>
> >> >>
> >> >> Modified: cfe/trunk/test/Driver/output-file-cleanup.c
> >> >> URL:
> >> >>
> >> >>
> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Driver/output-file-cleanup.c?rev=261774&r1=261773&r2=261774&view=diff
> >> >>
> >> >>
> >> >>
> ==============================================================================
> >> >> --- cfe/trunk/test/Driver/output-file-cleanup.c (original)
> >> >> +++ cfe/trunk/test/Driver/output-file-cleanup.c Wed Feb 24 15:49:28
> >> >> 2016
> >> >> @@ -38,6 +38,9 @@ invalid C code
> >> >> // RUN: test -f %t1.s
> >> >> // RUN: test ! -f %t2.s
> >> >>
> >> >> +// When given multiple .c files to compile, clang compiles them in
> >> >> order
> >> >> until
> >> >> +// it hits an error, at which point it stops.
> >> >> +//
> >> >> // RUN: touch %t1.c
> >> >> // RUN: echo "invalid C code" > %t2.c
> >> >> // RUN: touch %t3.c
> >> >> @@ -46,6 +49,6 @@ invalid C code
> >> >> // RUN: cd %T && not %clang -S %t1.c %t2.c %t3.c %t4.c %t5.c
> >> >> // RUN: test -f %t1.s
> >> >> // RUN: test ! -f %t2.s
> >> >> -// RUN: test -f %t3.s
> >> >> +// RUN: test ! -f %t3.s
> >> >> // RUN: test ! -f %t4.s
> >> >> -// RUN: test -f %t5.s
> >> >> +// RUN: test ! -f %t5.s
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> cfe-commits mailing list
> >> >> cfe-commits at lists.llvm.org
> >> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
> >> >
> >> >
> >
> >
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20160229/a47bee07/attachment-0001.html>
More information about the cfe-commits
mailing list