[llvm-dev] LLVM-based Mutation Testing, first results.

[Mehdi Amini via llvm-dev]

> On Jan 5, 2017, at 3:33 PM, Alex Denisov via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Hello, everybody.
> 
> We are working on a tool for mutation testing.

Nice!

> The work is still in progress and far away from being done.
> However, we have got some results already. And we would like to share them with you.
> But, let me give you a brief introduction first.
> 
> ### Mutation Testing
> 
> In a nutshell, Mutation Testing is a way to evaluate a quality of a test suite.
> 
> The approach suggests introducing a slight change into the code under a test (so called mutant).

It is not clear to me if you’re only changing the test itself, or the program the test exercise? (I’m more interested in the latter actually).

— 
Mehdi



> Then, if the test fails, mutant is considered as killed. Otherwise we say that mutant has survived.
> Along with that Mutation Testing introduces a metric, called Mutation Score:
> 
>  mutation score = (killed mutants) / (total mutants)
> 
> The higher the mutation score - the more adequate the test suite is. Hence, the more mutants are killed, the better.
> 
> Consider the following code:
> 
> void test() {
>  assert(sum(4, 5) > 0);
> }
> 
> int sum(int a, int b) {
> 	return a + b;
> }
> 
> Here, one can produce few mutants:
> 
> int sum(int a, int b) {
> 	return a * b;
> }
> 
> int sum(int a, int b) {
> 	return a;
> }
> 
> Both mutants will uncover that the test is not exhaustive enough.
> 
> Mutations
> 
> ### Our Results
> 
> We run our tool against two test targets from LLVM: IRTests and ADTTests. Based on our setup mutation score for them equals 43% and 66%, correspondingly.
> 
> We did not analyze all the results thoroughly, yet, but we plan to do so.
> At the very moment, it is not clear for us how to process the data more efficiently.
> Also, it is a bit more complicated, since we do not know the domain of LLVM internals very well.
> 
> Here I leave links to the results in case you are interested.
> 
> https://lowlevelbits.org/ADTTests/
> https://lowlevelbits.org/IRTests/
> 
> Source code of the tool itself is here:
> 
> https://github.com/mull-project/mull
> 
> We hope it helps to improve the quality of LLVM even further.
> Thanks for reading.
> We appreciate any feedback.
> 
> Best regards,
> Alex.
> -- 
> AlexDenisov
> Software Engineer, http://lowlevelbits.org
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev