[cfe-dev] [LLVMdev] LLVM / CLANG Test Infrastructure Question

[Chad Rosier]

On Jul 27, 2011, at 3:00 PM, Sundeep wrote:

> Hi All,
> 
> I am working on a bug in clang. I already have a fix for it and I am going
> through the "LLVM Testing Infrastructure Guide" to make sure I haven't
> broken anything else. I have few questions regarding the test suite
> infrastructure that I hope someone can answer.
> 
> 1. I checked out and built llvm, clang, and test-suite from svn tip. When I
> run llvm/test *without* my changes on x86, I see 3 failures. Is it expected?
> I was expecting svn tip on x86 to be clean. Is there a nightly regression
> result available somewhere that I can check for broken tests. Here is the
> report from my log:
> 
> Failing Tests (3):
>    Clang :: Analysis/iterators.cpp
>    Clang :: Driver/hello.c
>    Clang :: Preprocessor/macro_paste_c_block_comment.c
> 
>  Expected Passes    : 8757
>  Expected Failures  : 69
>  Unsupported Tests  : 551
>  Unexpected Failures: 3
> 

Because you're living life on the bleeding edge (i.e., pulling code from the svn repository) it's possible for this to happen from time to time.  A fix is probably already in the works.  One way to ensure you don't introduce a new error is to maintain a clean (i.e., unmodified version of llvm/clang), which you can use as a baseline.  Run regressions on both the clean and modified versions to see if you cause any additional failures.

> 2. I see clang has its own test suite under clang/test. I assume when I run
> llvm/test, the test infrastructure also runs clang/test.

How are you invoking the tests?  I typically do 'make check-all' from my build directory, which runs both regression suites.

> 3. For running projects/test-suite, the instruction guide directs to build
> llvm-gcc. Why is llvm-gcc required? Why is clang not being used?

Very possible the documentation needs updating.

> 4. I see many compile errors when running projects/test-suite. Is it
> expected?

No.

> 5. I used "make -i" to ignore compile erros and execute all tests under
> projects/test-suite. I finally get a report something like this:
> 
> Program                                                                  |
> GCCAS   Bytecode LLC compile LLC-BETA compile JIT codegen | GCC      CBE
> LLC     LLC-BETA JIT      | GCC/CBE GCC/LLC
> GMultiSource/Applications/Burg/burg                                       |
> 5.5484  115468   2.9722      *                3.0922      |  0.0100  *
> 0.0000 *         3.2400  | n/a     -       n
> MultiSource/Applications/ClamAV/clamscan                                 |
> 41.8306 1335184  34.4382     *                18.6692     |  0.1200  *
> 0.1300 *        19.5200  | n/a     0.92    n
> 
> I was also expecting to see a list of passing/failing test cases similar to
> llvm/test runs. Is there a make target to do so?
> 

You might try 'make -i report'

 Chad

> I apologies for asking so many questions. I followed directions from the
> "LLVM Testing Infrastructure Guide" but it doesn't address all these issues.
> I will appreciate your help to resolve these.
> 
> Thanks,
> Sundeep
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev