[llvm-dev] llvm-lit: 2>&1 and FileCheck

[David Blaikie via llvm-dev]

On Thu, Feb 23, 2017 at 6:54 AM Michael Kruse via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hi all,
>
> quite a few tests use the pattern "2>&1 | FileCheck %s". AFAIK how
> stdout and stderr are merged into a single character stream is
> undefined and depends e.g. on whether stdout is buffered. I think we
> are often saved by the fact that standard output is written only at
> the end of the program and stderr is unbuffered, i.e. always written
> before stdout.
>
> A lot of tests disable stdout using either "-o /dev/null" or
> "-disable-output", but not all. For instance,
> test/Transforms/SLPVectorizer/X86/reduction_unrolled.ll does not. It
> checks for output from stdout and stderr using the same FileCheck. The
> stderr it is checking even comes from -debug, which has an additional
> buffering layer (circular_raw_ostream).
>
> The testing guide [1] does not mention how to test stderr.
>
>
> My questions:
>
> 1. Are these tests, e.g. reduction_unrolled.ll fragile? Maybe I am
> missing something that says that interleaving stdout and stderr (and
> llvm::dbgs()) is well-defined in llvm-lit.
>
> 2. Can -debug (or -debug-only) be used in regression tests at all? I
> understood them as debugging aids only. I would not like if
> adding/changing DEBUG(dbgs() << ...); lines causing regression tests
> to fail.
>
>
Rough guesses, based on no broad review of test cases. All of this seems OK
except for the interleaved case(s) as you mentioned.

3f - generally it's probably best not to test for whether a source code
line has executed. That would make the test fragile - the observable
behavior should be tested instead. Though I would imagine it comes up
sometimes as the best thing to do in a bad situation.


> 3. What are the canonical ways to test...
> 3a) opt -stat output (e.g. "2>&1 | FileCheck\n; REQUIRES: asserts")

3b) A statistic from -stat being zero
> 3c) stderr only (and be sure that no lines from stdout will be
> interleaved with it)
> 3d) stdout and stderr at the same time, but independently.
> 3e) the output of DEBUG(dbgs() << ...) lines, if allowed to do so.
> 3f) If not, how to replace it? Eg. how to test whether a source code
> line has been executed.
>
> Thanks in advance,
> Michael
>
>
> [1] http://llvm.org/docs/TestingGuide.html
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170223/fd309aaa/attachment.html>