[LLVMdev] LLVM / CLANG Test Infrastructure Question
mcrosier at apple.com
Wed Jul 27 17:14:08 PDT 2011
On Jul 27, 2011, at 3:00 PM, Sundeep wrote:
> Hi All,
> I am working on a bug in clang. I already have a fix for it and I am going
> through the "LLVM Testing Infrastructure Guide" to make sure I haven't
> broken anything else. I have few questions regarding the test suite
> infrastructure that I hope someone can answer.
> 1. I checked out and built llvm, clang, and test-suite from svn tip. When I
> run llvm/test *without* my changes on x86, I see 3 failures. Is it expected?
> I was expecting svn tip on x86 to be clean. Is there a nightly regression
> result available somewhere that I can check for broken tests. Here is the
> report from my log:
> Failing Tests (3):
> Clang :: Analysis/iterators.cpp
> Clang :: Driver/hello.c
> Clang :: Preprocessor/macro_paste_c_block_comment.c
> Expected Passes : 8757
> Expected Failures : 69
> Unsupported Tests : 551
> Unexpected Failures: 3
Because you're living life on the bleeding edge (i.e., pulling code from the svn repository) it's possible for this to happen from time to time. A fix is probably already in the works. One way to ensure you don't introduce a new error is to maintain a clean (i.e., unmodified version of llvm/clang), which you can use as a baseline. Run regressions on both the clean and modified versions to see if you cause any additional failures.
> 2. I see clang has its own test suite under clang/test. I assume when I run
> llvm/test, the test infrastructure also runs clang/test.
How are you invoking the tests? I typically do 'make check-all' from my build directory, which runs both regression suites.
> 3. For running projects/test-suite, the instruction guide directs to build
> llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
Very possible the documentation needs updating.
> 4. I see many compile errors when running projects/test-suite. Is it
> 5. I used "make -i" to ignore compile erros and execute all tests under
> projects/test-suite. I finally get a report something like this:
> Program |
> GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE
> LLC LLC-BETA JIT | GCC/CBE GCC/LLC
> GMultiSource/Applications/Burg/burg |
> 5.5484 115468 2.9722 * 3.0922 | 0.0100 *
> 0.0000 * 3.2400 | n/a - n
> MultiSource/Applications/ClamAV/clamscan |
> 41.8306 1335184 34.4382 * 18.6692 | 0.1200 *
> 0.1300 * 19.5200 | n/a 0.92 n
> I was also expecting to see a list of passing/failing test cases similar to
> llvm/test runs. Is there a make target to do so?
You might try 'make -i report'
> I apologies for asking so many questions. I followed directions from the
> "LLVM Testing Infrastructure Guide" but it doesn't address all these issues.
> I will appreciate your help to resolve these.
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
More information about the llvm-dev