[LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

[Haghighat, Mohammad R]

Hi Karen,

Thanks much for your comment and for sharing of your experience. icFuzz has a core that is "really" random, but does not cover the entire C space. The tool was designed from scratch to be extensible, and comes with a couple of extensions that target some of compiler optimizations optimizing compilers typically do: CSE, loop interchange, vectorization, etc. But even in the case of extensions, other than the structure of the extension, most of the details are highly parametric and configurable. The chances of a totally random test meeting all the criteria for certain optimizations to kick in is lower than the case in guided test-generation. I actually have the x-y charts where x is the number of generated tests and y is the number of fails for both completely random tests as well as random+guided tests, and the curve of guided tests is significantly above that of random tests. Also, the test generator is restricted to the "unsigned int" type where the C++ semantics are precisely defined even in the cases of overflow. Also, features that have implementation-dependent behavior are excluded by design (e.g., expressions including side effects during their evaluation, array out of bounds access, etc.).

But you are right in that the generated code has some structure ;) 

Cheers,
-moh

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Karen Shaeffer
Sent: Monday, June 24, 2013 5:17 PM
To: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

On Mon, Jun 24, 2013 at 11:20:09PM +0000, Haghighat, Mohammad R wrote:
> The tests by design are syntactically correct, semantically correct, and have deterministic output.
> 
> -moh

Hi,
I wanted to believe you had a randomized code generator that could cover the entire
valid input space. (smiles ;) But the test cited was not generated by a randomized
code generator. It is way too structured. Even so, it could have been generated by a
partially randomized code generator that uses valid code idioms with some aspects
randomized. Interesting to think about it.

I wrote a fully randomized Java bytecode sequence generator once. It covered the entire
valid input space. I used it to test a Java hardware accelerator chip that had just been
taped out. It found quite a few bugs, where 4 of them were fatal to the chip.

Writing such a tool for a general purpose compiler would be quite a bit more
difficult, but it is doable. Imagine releasing compilers that are bug free,
at least for a given language and a particular platform. It's doable, and I wonder
why corporations don't invest in it.

enjoy,
Karen
-- 
Karen Shaeffer
Neuralscape, Mountain View, CA 94040
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev