[LLVMdev] LLVM / CLANG Test Infrastructure Question

[Sundeep]

Hi Chad,

I finally got it to work. I think "LLVM Testing Infrastructure Guide" should
be updated to reflect the latest changes (use clang instead of llvm-gcc and
make command etc.).

I ran into problems with using system headers and startup files (crt*.o). So
I had to build and install llvm-gcc, next build clang using headers from
llvm-gcc install dir, and then use clang to run regressions under
projects/test-suite. It works beautifully.

Anyways, I wanted to thank you for all your help. I really appreciate it.

-Sundeep

> -----Original Message-----
> From: Chad Rosier [mailto:mcrosier at apple.com]
> Sent: Thursday, July 28, 2011 12:37 PM
> To: Sundeep
> Cc: cfe-dev at cs.uiuc.edu; 'LLVM Dev'
> Subject: Re: [LLVMdev] LLVM / CLANG Test Infrastructure Question
> 
> 
> On Jul 28, 2011, at 8:18 AM, Sundeep wrote:
> 
> >>> 1. I checked out and built llvm, clang, and test-suite from svn
> tip.
> >> When I
> >>> run llvm/test *without* my changes on x86, I see 3 failures. Is it
> >> expected?
> >>> I was expecting svn tip on x86 to be clean. Is there a nightly
> >> regression
> >>> result available somewhere that I can check for broken tests. Here
> is
> >> the
> >>> report from my log:
> >>>
> >>> Failing Tests (3):
> >>>   Clang :: Analysis/iterators.cpp
> >>>   Clang :: Driver/hello.c
> >>>   Clang :: Preprocessor/macro_paste_c_block_comment.c
> >>>
> >>> Expected Passes    : 8757
> >>> Expected Failures  : 69
> >>> Unsupported Tests  : 551
> >>> Unexpected Failures: 3
> >>>
> >>
> >> Because you're living life on the bleeding edge (i.e., pulling code
> >> from the svn repository) it's possible for this to happen from time
> to
> >> time.  A fix is probably already in the works.  One way to ensure
> you
> >> don't introduce a new error is to maintain a clean (i.e., unmodified
> >> version of llvm/clang), which you can use as a baseline.  Run
> >> regressions on both the clean and modified versions to see if you
> cause
> >> any additional failures.
> >
> > Makes sense.
> >
> >>> 2. I see clang has its own test suite under clang/test. I assume
> when
> >> I run
> >>> llvm/test, the test infrastructure also runs clang/test.
> >>
> >> How are you invoking the tests?  I typically do 'make check-all'
> from
> >> my build directory, which runs both regression suites.
> >
> > Yes, I am doing "make check-all".
> >
> >>> 3. For running projects/test-suite, the instruction guide directs
> to
> >> build
> >>> llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
> >>
> >> Very possible the documentation needs updating.
> >
> > I see. Do you run projects/test-suite with clang?
> >
> 
> From your build/project/test-suite directory, try running:
> 
> make ARCH=x86_64 CC_UNDER_TEST_IS_CLANG=1
> CC_UNDER_TEST_TARGET_IS_X86_64=1 DISABLE_CBE=1 DISABLE_JIT=1
> ENABLE_HASHED_PROGRAM_OUTPUT=1 ENABLE_OPTIMIZED=1 LD_ENV_OVERRIDES="env
> DYLD_LIBRARY_PATH=/path/to/your/build/Release+Asserts/lib"
> LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1
> TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64"
> TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang
> TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++
> TEST=simple USE_REFERENCE_OUTPUT=1 report
> 
> ***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure
> with '--enable-optimized' at config time (i.e., ../llvm/configure --
> enable-optimized).
> 
> You may have to fiddle with the options, but I think this will get you
> going in the right direction.
> 
> >>> 4. I see many compile errors when running projects/test-suite. Is
> it
> >>> expected?
> >>
> >> No.
> >
> > I will try to debug. Do you have a log file from your previous runs
> that I
> > can use?
> >
> 
> I don't have any on hand, nor do I think they would be of help.  Try
> the above make command a see what happens.  For the most part
> everything should PASS.
> 
> > Thanks Chad. I appreciate it.
> >
> 
> You're welcome Sundeep.
> 
> > -Sundeep
> >