[LLVMdev] LLVM / CLANG Test Infrastructure Question

[Chad Rosier]

On Jul 28, 2011, at 8:18 AM, Sundeep wrote:

>>> 1. I checked out and built llvm, clang, and test-suite from svn tip.
>> When I
>>> run llvm/test *without* my changes on x86, I see 3 failures. Is it
>> expected?
>>> I was expecting svn tip on x86 to be clean. Is there a nightly
>> regression
>>> result available somewhere that I can check for broken tests. Here is
>> the
>>> report from my log:
>>> 
>>> Failing Tests (3):
>>>   Clang :: Analysis/iterators.cpp
>>>   Clang :: Driver/hello.c
>>>   Clang :: Preprocessor/macro_paste_c_block_comment.c
>>> 
>>> Expected Passes    : 8757
>>> Expected Failures  : 69
>>> Unsupported Tests  : 551
>>> Unexpected Failures: 3
>>> 
>> 
>> Because you're living life on the bleeding edge (i.e., pulling code
>> from the svn repository) it's possible for this to happen from time to
>> time.  A fix is probably already in the works.  One way to ensure you
>> don't introduce a new error is to maintain a clean (i.e., unmodified
>> version of llvm/clang), which you can use as a baseline.  Run
>> regressions on both the clean and modified versions to see if you cause
>> any additional failures.
> 
> Makes sense.
> 
>>> 2. I see clang has its own test suite under clang/test. I assume when
>> I run
>>> llvm/test, the test infrastructure also runs clang/test.
>> 
>> How are you invoking the tests?  I typically do 'make check-all' from
>> my build directory, which runs both regression suites.
> 
> Yes, I am doing "make check-all".
> 
>>> 3. For running projects/test-suite, the instruction guide directs to
>> build
>>> llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
>> 
>> Very possible the documentation needs updating.
> 
> I see. Do you run projects/test-suite with clang?
> 

>From your build/project/test-suite directory, try running:

make ARCH=x86_64 CC_UNDER_TEST_IS_CLANG=1 CC_UNDER_TEST_TARGET_IS_X86_64=1 DISABLE_CBE=1 DISABLE_JIT=1 ENABLE_HASHED_PROGRAM_OUTPUT=1 ENABLE_OPTIMIZED=1 LD_ENV_OVERRIDES="env DYLD_LIBRARY_PATH=/path/to/your/build/Release+Asserts/lib" LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1 TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64" TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++ TEST=simple USE_REFERENCE_OUTPUT=1 report

***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure with '--enable-optimized' at config time (i.e., ../llvm/configure --enable-optimized).

You may have to fiddle with the options, but I think this will get you going in the right direction.

>>> 4. I see many compile errors when running projects/test-suite. Is it
>>> expected?
>> 
>> No.
> 
> I will try to debug. Do you have a log file from your previous runs that I
> can use?
> 

I don't have any on hand, nor do I think they would be of help.  Try the above make command a see what happens.  For the most part everything should PASS.

> Thanks Chad. I appreciate it.
> 

You're welcome Sundeep.

> -Sundeep
>