Automating test reuse for highly configurable software

Authors Stefan Fischer
Rudolf Ramler
Lukas Linsbauer
Alexander Egyed
Title Automating test reuse for highly configurable software
Booktitle Proceedings of the 23rd International Systems and Software Product Line Conference (SPLC 2019)
Type in proceedings
Publisher ACM
Volume A
ISBN 978-1-4503-7138-4
DOI 10.1145/3336294.3336305
Month September
Year 2019
Pages 1-11
SCCH ID# 19043

Dealing with highly configurable systems is generally very complex. Hundreds of different analysis techniques have been conceived to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This is challenging due to the large number of possible configurations and because tests themselves are rarely configurable and instead built for specific configurations. Existing tests can usually not be reused on other configurations. Therefore, tests need to be adapted for the specific configuration they are supposed to test. In this paper we report on an experiment about reusing tests in a configurable system. We used manually developed tests for specific configurations of Bugzilla and investigated which of them could be reused for other configurations. Moreover, we automatically generated new test variants (by automatically reusing from existing ones) for combinations of previous configurations. Our results showed that we can directly reuse some tests for configurations which they were not intended for. Nonetheless, our automatically generated test variants generally yielded better results. When applying original tests to new configurations we found an average success rate for the tests of 81,84%. In contrast, our generated test variants achieved an average success rate of 98,72%. This is an increase of 16,88%.