[llvm-dev] llvm-lit: 2>&1 and FileCheck

[Daniel Berlin via llvm-dev]

On Thu, Feb 23, 2017 at 10:48 AM, Robinson, Paul via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>
>
> > -----Original Message-----
> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> > Michael Kruse via llvm-dev
> > Sent: Thursday, February 23, 2017 6:53 AM
> > To: llvm-dev
> > Subject: [llvm-dev] llvm-lit: 2>&1 and FileCheck
> >
> > Hi all,
> >
> > quite a few tests use the pattern "2>&1 | FileCheck %s". AFAIK how
> > stdout and stderr are merged into a single character stream is
> > undefined and depends e.g. on whether stdout is buffered. I think we
> > are often saved by the fact that standard output is written only at
> > the end of the program and stderr is unbuffered, i.e. always written
> > before stdout.
> >
> > A lot of tests disable stdout using either "-o /dev/null" or
> > "-disable-output", but not all. For instance,
> > test/Transforms/SLPVectorizer/X86/reduction_unrolled.ll does not. It
> > checks for output from stdout and stderr using the same FileCheck. The
> > stderr it is checking even comes from -debug, which has an additional
> > buffering layer (circular_raw_ostream).
> >
> > The testing guide [1] does not mention how to test stderr.
> >
> >
> > My questions:
> >
> > 1. Are these tests, e.g. reduction_unrolled.ll fragile? Maybe I am
> > missing something that says that interleaving stdout and stderr (and
> > llvm::dbgs()) is well-defined in llvm-lit.
>
> I'd consider them fragile, but obviously their behavior has been
> consistent across a variety of bots for some time.  So the fragility
> is a bit pedantic/theoretical.  "The behavior is undefined but I know
> what I'm doing!"
>
> There are times running a test when I've seen interleaved stdout/stderr
> text, but not the text that a CHECK was looking for; so I think people
> are getting lucky in at least some cases.
>
> >
> > 2. Can -debug (or -debug-only) be used in regression tests at all? I
> > understood them as debugging aids only. I would not like if
> > adding/changing DEBUG(dbgs() << ...); lines causing regression tests
> > to fail.
>
> The line between "debugging aid" and "event logging" is not clear, but
> I have written tests relying on logging-style output; I think that's ok.
> As always you want your CHECKs to be specific enough to avoid false
> matches but not so specific that they become too fragile.
>

Right.
Where we can, we prefer to have separate printing passes, but that only
really works well for analysis and preparation transformations.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170223/99de6240/attachment.html>