[cfe-dev] [Clang SA] Towards test generation

[Bhargava Shastry via cfe-dev]

Hi,

Clang SA is a great tool for discovering defects. From a program testing
perspective, it would be nice to leverage it for test case
generation. A first step could be to make Clang
SA spit out path constraints alongside bug reports, and use a solver to
generate (synthesize) test input.

Has anyone else toyed with this idea or fancies having a go? Pointers to
previous work in this direction much appreciated :-)

I am aware that KLEE does exactly this using full-program SE but sadly,
it hasn't been demonstrated on anything bigger than coreutils. Since
Clang SA performs under-constrained SE, it scales up against large
codebases, so it is a good candidate to tease out test cases for more
complex programs. Naturally, at some point under-constraining may prove
an obstacle for program input generation but the point is to at least
get to a point where we can focus on the problem of ``extrapolating"
under-constrained input to program input.

-----
A simple example to demonstrate the use-case:

1. #include <stdio.h>
2. int main() {
3.  int i, j;
4.  scanf("%d," &i);
5.  j = 10/i;
6.  return j;
7. }

Clang SA says div-by-zero possible on line 5. Additionally, it outputs
the constraint (i==0), which is trivially sat for i equals zero. Zero is
the synthesized test input. Of course, this is a very simple example
where the defect is obvious to a careful reviewer but constraints deep
within a program might be hard to reason manually.
----

Best,
Bhargava