[LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

[Haghighat, Mohammad R]

> Is it feasible to set up runs with different flags?

Absolutely. In fact, a common way of using tool is to compare two different settings of two different compilers (e.g., MSVC -O0 vs. ICC -O3, etc.). I am currently working with the Mozilla folks on Emscripten/asm.js. So the way I now use icFuzz is comparing "clang -O2" vs. "emcc -O2 (which compiles C++ to JS) and running the generated JavaScript code which generates the final output" :)

> Are there non-floating-point cases?

No. Currently, icFuzz tests are restricted to the "unsigned int" type to avoid FP rounding errors and be absolutely false positive.

-----Original Message-----
From: Hal Finkel [mailto:hfinkel at anl.gov] 
Sent: Wednesday, June 26, 2013 10:44 AM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

----- Original Message -----
> Great job, Hal!
> 
> Sure. I'd be happy to run icFuzz and report the fails once these bugs
> are fixed and thereafter whenever people want new runs. Obviously,
> this can be automated, but the problem is that icFuzz is not
> currently open sourced.

I would be happy to see this open sourced, but I think that we can work something out regardless.

Also, once we get the current set of things resolved, I think it would be useful to test running with:

 -- -O3, LTO (-O4 or -flto),
 -- -fslp-vectorize, -fslp-vectorize-aggressive (which are actually separate optimizations)
 -- -ffast-math (if you can do floating point with tolerances, or at least -ffinite-math-only), -fno-math-errno
 (and there are obviously a whole bunch of non-default code-generation and target options).

Is it feasible to set up runs with different flags?

> Once there's a bug in the compiler, there's
> really no limit in the number of failing tests that can be
> generated, so it's more productive to run the generator after the
> previously reported bugs are fixed.

Agreed.

> 
> We've also seen cases where the results of "clang -O2" are different
> on Mac vs. Linux/Windows.

I recall an issue related to default settings for FP, and differences with libm implementation. Are there non-floating-point cases?

> 
> Just let me know when you want a new run.

Will do!

 -Hal

> 
> Cheers,
> -moh
> 
> -----Original Message-----
> From: Hal Finkel [mailto:hfinkel at anl.gov]
> Sent: Wednesday, June 26, 2013 7:35 AM
> To: Haghighat, Mohammad R
> Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
> Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly
> generated tests that fail on clang 3.4 trunk
> 
> ----- Original Message -----
> > 
> > Hi Moh,
> > 
> > 
> > Thanks for this. I’m really glad to see the work you’re doing in
> > this
> > area and believe it will be extremely helpful in improving the
> > quality of the compiler.
> > 
> > 
> > -Jim
> > 
> > 
> > 
> > On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> > mohammad.r.haghighat at intel.com > wrote:
> > 
> > 
> > 
> > 
> > 
> > Hi,
> > 
> > I just submitted a bug report with a package containing 107 small
> > test cases that fail on the latest LLVM/clang 3.4 main trunk
> > (184563). Included are test sources, compilation commands, test
> > input files, and results at –O0 and –O2 when applicable.
> > 
> > http://llvm.org/bugs/show_bug.cgi?id=16431
> > 
> > These tests have been automatically generated by an internal tool
> > at
> > Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> > very small. For example, for the following simple loop (test t5702)
> > on MacOS X, clang at –O2 generates a binary that crashes:
> > 
> > // Test Loop Interchange
> > for (j = 2; j < 76; j++) {
> > for (jm = 1; jm < 30; jm++) {
> > h[j-1][jm-1] = j + 83;
> > }
> > }
> > 
> > The tests are put in to two categories
> > - tests that have different runtime outputs when compiled at -O0
> > and
> > -O2 (this category also includes runtime crashes)
> > - tests that cause infinite loops in the Clang optimizer
> > 
> > Many of these failing tests could be due to the same bug, thus a
> > much
> > smaller number of root problems are expected.
> > 
> > Any help with triaging these bugs would be highly appreciated.
> 
> I've gone through all of the miscompile cases, used bugpoint to
> reduce them, and opened individual PRs for several distinct bugs. So
> far we have: PR16455 (loop vectorizer), PR16457 (sccp), PR16460
> (instcombine). Thanks again for doing this! Do you plan on repeating
> this testing on a regular basis? Can it be automated?
> 
>  -Hal
> 
> > 
> > Thanks,
> > -moh
> > 
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory