Identifying mutant hierarchies to reduce test effort in mutation testing

Abstract

Mutation testing is a fault-based testing approach. The main goal of mutation testing is to assess the quality of a test suite, and in that process, test the software. The approach asserts the absence of pre-specified faults. In mutation testing, a single syntactic change is made and the resulting program is referred to as ‘Mutant’. The mutants obtained by making n syntactic changes are known as nth-order mutants’. Mutation Testing is based on two hypotheses, (i) Coupling Hypothesis assumes if a test suite can detect a first-order mutant from the original program, it will detect most higher-order mutants, (ii) Competent Programmer Hypothesis assumes the program to be tested is “close” to the intended program. Mutation testing is practically infeasible because the number of generated mutants is myriad for each program. It is also resource intensive requiring substantial amount of computation power and time to execute all the mutants against the original program for a given test suite.

The objective of this thesis is two-fold. First, an empirical study is done to validate a formal approach proposed [Kapoor, K and Bowen, J. P, 2004] to reduce the test effort required in mutation testing. The formal approach orders the mutants in a hierarchy such that a mutant occurring higher in the hierarchy is stronger than a mutant occurring lower in the hierarchy. If a test suite can distinguish the original program from a stronger mutant, it is guaranteed to distinguish any mutant occurring lower than the stronger mutant in the hierarchy. Second, the concept of conditional strong mutant for relational operators is proposed which is a specialisation of the strong mutant concept. It further reduces the test effort by identifying more number of strong mutants through some constraints on the operands of the relational operators.

The savings obtained by using conditional strong mutants is more than that obtained by strong mutants. There is a saving of 80% in the test effort required involving mutants generated by invoking the Relational Operator Replacement (ROR) operator on conditions present in the branch statements of a program. The analysis done applies to any program containing relational operators in the conditions of branch statements.