[llvm-dev] LLVM-based Mutation Testing, first results.

[Alex Denisov via llvm-dev]

Hello, everybody.

We are working on a tool for mutation testing. The work is still in progress and far away from being done.
However, we have got some results already. And we would like to share them with you.
But, let me give you a brief introduction first.

### Mutation Testing

In a nutshell, Mutation Testing is a way to evaluate a quality of a test suite.

The approach suggests introducing a slight change into the code under a test (so called mutant). Then, if the test fails, mutant is considered as killed. Otherwise we say that mutant has survived.
Along with that Mutation Testing introduces a metric, called Mutation Score:

  mutation score = (killed mutants) / (total mutants)
  
The higher the mutation score - the more adequate the test suite is. Hence, the more mutants are killed, the better.

Consider the following code:

void test() {
  assert(sum(4, 5) > 0);
}

int sum(int a, int b) {
	return a + b;
}

Here, one can produce few mutants:

int sum(int a, int b) {
	return a * b;
}

int sum(int a, int b) {
	return a;
}

Both mutants will uncover that the test is not exhaustive enough.

Mutations

### Our Results

We run our tool against two test targets from LLVM: IRTests and ADTTests. Based on our setup mutation score for them equals 43% and 66%, correspondingly.

We did not analyze all the results thoroughly, yet, but we plan to do so.
At the very moment, it is not clear for us how to process the data more efficiently.
Also, it is a bit more complicated, since we do not know the domain of LLVM internals very well.

Here I leave links to the results in case you are interested.

https://lowlevelbits.org/ADTTests/
https://lowlevelbits.org/IRTests/

Source code of the tool itself is here:

https://github.com/mull-project/mull

We hope it helps to improve the quality of LLVM even further.
Thanks for reading.
We appreciate any feedback.

Best regards,
Alex.
-- 
AlexDenisov
Software Engineer, http://lowlevelbits.org