Evolutionary approach for combinatorial testing of software product lines

Software Product Line (SPL) focuses on common features reusability, formulated for different software products. Complete testing on the entire SPL is known to be unfeasible. This is due to the very large number of possible products to be produced; configured using all possible sets of features in the SPL. Combinatorial Testing (CT) has been widely used in software testing as it is able to generate test input for a single software product that deviates from exhaustive testing, nevertheless proven to be effective. In SPL testing, to generate minimal test configuration that maximizes t-wise coverage is not trivial, especially when dealing with a huge number of features and when constraints have to be satisfied, which is the case in most SPL systems. Two salient obstacles in SPL testing are identified; (1) time required to cover feature configuration testing for large scale Feature Models and higher strength of t-wise testing, and (2) insufficient coverage of feature interaction towards higher fault detection due to uniform strength of feature combination. In the case of unfeasible higher strength CT-based testing for SPL, this thesis proposed a relaxed version of Covering Array (CA), i.e. Variable-strength CA (VCA). In highly configurable systems, the entire configuration spaces contain several groups of sub-configurations that have different level of risks that can cause failures. By focusing on these group of subconfigurations, the space of testing artefacts can be reduced, while still maintain the results of testing. Practical Combinatorial Testing is challenging, such that it is hard to decide what level of interaction strength to be applied to which different groups of features. There are two inherent challenges; (1) to decide what value of interaction strength one should apply, and (2) to choose which group or set of features the interaction strength should be applied to. For these reasons, part of this thesis presents and evaluates a new technique aimed at tackling all the mentioned difficulties in the combinatorial testing of SPL system. The approaches are (1) adopting an Estimation of Distribution Algorithm approach to aid the construction of a covering array driven by second order feature dependence, and (2) construct a feature configuration dependence graph to assist in building the variable-strength covering array. These techniques are implemented as COTED (Combinatorial Testing using Bivariate EDA). Experiments have been done to gauge the performance of COTED from three different facets; (1) the ability to reduce test configuration redundancy, (2) the ability to achieve better rate of interaction coverage and (3) the ability to detect high number of faults. Four empirical studies conducted to compare COTED against other approaches (namely ICPL, LOOKUP and CASA). The statistical analysis of the results obtained from the experiments are promising. For the first facet, COTED is comparable with LOOKUP, but a slightly better redundancy is achieved by ICPL with significant level of 0.01. For the second facet, COTED able to outperform ICPL and LOOKUP with significance level of 0.1. For the third facet, the two sets of VCA test configurations (i.e. (i) 2-wise and partial 3-wise, and (ii) 2-wise, partial 3 and partial 4-wise) generated by COTED are able to achieve competitive fault detection rate (with significance level of 0.05) using less number of test configurations. The result shows that COTED increase the interaction coverage rate, and managed to produce less number of test configurations to detect an almost same number of faults as the other strategies, hence, suggests that it is able to improve Combinatorial Interaction Testing (CIT) for SPL.

Text FSKTM 2020 14 IR.pdf Download (1MB)

Actions (login required)

View Item