Why software testing fails: Common pitfalls observed in a critical smart metering project

• Stefan Mohacsi
• Rudolf Ramler

• Dietmar Winkler
• Stefan Biffl
• Johannes Bergsmann

Over the last decades a considerable share of software engineering research has been dedicated to the area of software testing. Still, however, test-ing often fails or causes major problems in practice. In this paper we present in-sights and experiences from a large project in the energy sector. The obligatory switch from analog energy meters to smart metering technology poses a big challenge for many energy providers. Apart from technical issues concerning meters and transmission technology, the adaption of the internal business pro-cesses together with the development of backend software can turn out to be more difficult than expected. The criticality, size and complexity of the ana-lyzed project are reflected in software and system testing, where the underesti-mated effort, mistakes, and wrong decisions caused serious difficulties. In our work we describe the observed testing problems and the underlying causes. Subsequently, we compare the identified problems with a catalogue of com-monly known testing pitfalls and anti-patterns. The results show that the majori-ty of the observed problems are not new or specific to the studied project. Fur-thermore, additional candidates for extending the list of common pitfalls are identified. Besides recommendations on how to mitigate the problems in the studied project, we conclude with the general insight that there is a great poten-tial to improve software testing practice by developing measures for early recognition, communication, and avoiding of common mistakes.