[llvm-dev] llvm-lit: 2>&1 and FileCheck

[Michael Kruse via llvm-dev]

Hi all,

quite a few tests use the pattern "2>&1 | FileCheck %s". AFAIK how
stdout and stderr are merged into a single character stream is
undefined and depends e.g. on whether stdout is buffered. I think we
are often saved by the fact that standard output is written only at
the end of the program and stderr is unbuffered, i.e. always written
before stdout.

A lot of tests disable stdout using either "-o /dev/null" or
"-disable-output", but not all. For instance,
test/Transforms/SLPVectorizer/X86/reduction_unrolled.ll does not. It
checks for output from stdout and stderr using the same FileCheck. The
stderr it is checking even comes from -debug, which has an additional
buffering layer (circular_raw_ostream).

The testing guide [1] does not mention how to test stderr.


My questions:

1. Are these tests, e.g. reduction_unrolled.ll fragile? Maybe I am
missing something that says that interleaving stdout and stderr (and
llvm::dbgs()) is well-defined in llvm-lit.

2. Can -debug (or -debug-only) be used in regression tests at all? I
understood them as debugging aids only. I would not like if
adding/changing DEBUG(dbgs() << ...); lines causing regression tests
to fail.

3. What are the canonical ways to test...
3a) opt -stat output (e.g. "2>&1 | FileCheck\n; REQUIRES: asserts")
3b) A statistic from -stat being zero
3c) stderr only (and be sure that no lines from stdout will be
interleaved with it)
3d) stdout and stderr at the same time, but independently.
3e) the output of DEBUG(dbgs() << ...) lines, if allowed to do so.
3f) If not, how to replace it? Eg. how to test whether a source code
line has been executed.

Thanks in advance,
Michael


[1] http://llvm.org/docs/TestingGuide.html