Mutation Testing

About

Mutation testing assesses software test quality by adding small code defects and checking if tests can find them. It measures how well tests identify errors, aiming for a high mutation detection rate for a strong test suite.

A mutation is nothing but a single syntactic change made in the code. Each mutant code should differ from the original code by one mutation.

Glossary

mutant

a unit modification of the code (e.g: != replaced by ==)

killed/captured

a mutant was killed if the unit test fails (positive outcome)

lived/escaped

a mutant escapes if the unit test doesn’t fail (negative outcome)

uncovered

a mutant is uncovered if no test covers the mutated code

Code coverage

Code coverage is a false indicator of protection. Good coverage does not imply good tests. It is important to distinguish line and branch test coverage:

• Line Coverage - the percent of lines executed by this test run

• Branch Coverage - the percent of branches executed by this test run

Example:

public int getNameLength(boolean isAdminUser) {
    User user = null;
    if (isAdminUser) {
        user = new John();
    }
    return user.getName().length();
}

• if we call the getNameLength method with isAdminUser set to true, we will get 100% line coverage

• branch coverage is 50% - we can see there is something missing in our testing

• NullPointerException in case isAdminUser flag is false

Coverage runner in IntelliJ

IntelliJ's Coverage runner allows you to switch from default Line Coverage to JaCoCo for collecting Branch Coverage.

Types of Mutation Testing

Mutation testing is categorised into 3 types:

• statement mutation – developer cut and pastes a part of a code of which the outcome may be a removal of some lines

• value mutation – values of primary parameters are modified

• decision mutation – control statements are to be changed

What should be changed in Mutant Program?

• operand replacement

  • replace the operand with another operand or with the constant value

  • if(x>y) replace x and y values

• expression Modification

  • replace an operator or insert a new operator in a statement

  • if(x==y) replace == with >=

• statement modification

  • statements are modified to create mutant programs

  • delete the else part in an if-else statement

  • delete the entire if-else statement to check how a program behaves

PIT

PIT mutation testing runs unit tests against automatically modified code, providing a Mutation Coverage metric. It is fast, easy to use (e.g., Maven, Gradle), actively developed, and supported. The reports from PIT combine line and mutation coverage information.