Mutation testing with PHP

In this post we are going to see what mutation test is and how to apply this technique in PHP.

What is Mutation Testing?

Mutation testing is a testing technique that highlight hidden issues sometimes impossible to identify through a conventional test mechanism.
In a practical way, let’s say we have a portion of code correctly covered with a unit test (i.e., phpunit test). The mutation test process first changes, “mutates” with small variation a portion of code, that usually pass the unit test, then checks if the variation correctly will be “killed”; detected as a unit test failure.

Below a very simple example.
From this original code:

if ($a && $b) {
    $c = true;
} else {
    $c = false;
}

We can mutate the condition (if) and produce the following mutant:

if ($a || $b) {
    $c = true;
} else {
    $c = false;
}

To “kill” the mutant the test should follow a RIP model:

1. Reachability: the test should reach the mutated statement;
2. Infection: the test should raise an incorrect status;
3. Propagation: the failure should be propagated to the test output;

A strong kill will satisfy all of the above 3 statements; a weak kill won’t reach the propagation level.

Types of Mutation testing

A mutation can be based on:

• value (ie: increase a counter value);
• decision (ie: change an if statement condition from == to !=);
• statement (ie: delete or duplicate a line);

Mutation test on PHP

At the moment, the only valid mutation test framework for PHP is Humbug.
It does an initial test run and saves the result, then tests the classes running mutation tests. The output is pretty complete with a score indicator, code coverage and other useful info.

You can find more info about Humbug in the official GitHub repo.