Gremlins

Tests Go Report Card Maintainability Codacy Badge codecov

WARNING: Gremlins is in its early stages of development, and it can be unstable, poorly performant and not really polished. At the moment, Gremlins runs in the project directory, which means it modifies the original files in place. It should not break anything, but it’s highly probable it will modify the formatting of the source code. Be sure to have a backup of your project before running Gremlins on it.

Gremlins is a mutation testing tool for Go.

What is Mutation Testing

Code coverage unreliable to measure test quality. It is too easy to have tests that exercise a piece of code but don’t test anything at all. Mutation testing works by mutating the code exercised by the tests and verifying if the mutation is caught by the test suite. Imagine gremlins going into your code and messing around: will your test suit catch their damage?

Here is a nice intro to mutation testing.

How to use Gremlins

To execute a mutation test run, from the root of a Go module execute:

$ gremlins unleash

Gremlins only tests mutations of parts of the code already covered by test cases. If a mutant is not covered, why bother testing? You already know it will not be caught. In any case, Gremlins will report which mutations aren’t covered.

If the Go test run needs build tags, they can be passed along:

$ gremlins unleash --tags "tag1,tag2"

To perform the analysis without actually running the tests:

$ gremlins unleash --dry-run

Supported mutations

Conditionals Boundaries

Original Mutated
> >=
>= >
< <=
<= <

Example:

if a > b {
  // Do something
}

will be changed to

if a < b {
  // Do something
}

Conditionals Negation

Original Mutated
== !=
!= ==
> <=
<= >
< >=
>= <

Example:

if a == b {
  // Do something
}

will be changed to

if a != b {
  // Do something
}

Increment Decrement

Original Mutated
++
++

Example:

func incr(i int) int
  return i++
}

will be changed to

func incr(i int) int {
  return i--
}

Invert Negatives

It will invert negative numbers.

Example:

func negate(i int) int {
  return -i
}

will be changed to

func negate(i int) int {
  return +i
}

Arithmetic Base

Original Mutated
+
+
* /
/ *
% *

Example:

a := 1 + 2

will be changed to

a := 1 - 2

Current limitations

There are some limitations on how Gremlins works right now, but rest assured we’ll try to make things better.

  • Gremlins can be run only from the root of a Go module and will run all the test suite. This is a problem if the tests are especially slow.
  • For each mutation, Gremlins will run all the test suite. It would be better to only run the test cases that actually cover the mutation.
  • Gremlins doesn’t support custom test commands; if you have to do anything different from go test ./... to run your test suite, most probably it will not work with Gremlins.
  • There is no way to implement custom mutations.
  • It is not tested on Windows as of now and most probably it will not work there.

What inspired Gremlins

Mutation testing exists since the early days of computer science, so loads of papers and articles do exists. Gremlins is inspired from those.

Among the existing mutation testing tools, Gremlins is inspired especially by PITest.

Other Mutation Testing tools for Go

There is not much around, except from:

Contributing

See contributing.

GitHub

View Github