Test smarter, not harder!
Test management for complex software systems
The steadily growing scope and rising complexity of software systems has caused an explosion in testing effort. Instead of escalating testing effort or hoping for better test automation tools, the key to effective and efficient testing in many cases is in focusing testing effort on the defective system components. The distribution of defects over a software system is typically not uniform. In a research project at SCCH, experts from different fields are working on methods to enable identification of the defective components of a system before testing.
Overview
Software systems are continuously growing. A new version is accompanied by new features. Each new software version on the market is bigger and more complex than its predecessor. However, the increasing scope also increases the necessary effort for quality assurance and testing – and disproportionally to the number of new features. Each new version requires the assurance that the new features are free of defects, but also that existing functionality is maintained despite enhancements. The attempt to completely test the overall system with each new version is a technical and economic challenge.
Challenge
A promising solution is to test exactly the locations that actually contain defects, which amounts to testing correctly. Even for software systems, the Pareto principle applies: 80% of the errors in a software system reside in only 20% of the components, while 50% of the components are error-free. Yet how can we determine which components are defective? And how can we acquire this information before testing? Data mining technology detects recurring patterns and interrelations in data; these can then be applied in a model to predict future events.
Solution
To predict defects in software systems, data are taken from version management, error and release management, and static analysis of source code. In multiple studies, SCCH has explored the datasets of large industrial software systems and identified typical patterns and their relation to defects. Typical patterns include the type or number of changes that were made in a software component. Each pattern provides indicators of whether a software component is defective. The combination of many such patterns in the form of a predictive model enables a prognosis of defects in the software components of a future version. "Currently we achieve the best results with data from static analysis of source code and change statistics. The precision of prognosis is currently around 80%: in 8 of 10 cases, we can determine before testing whether a component is defective," describes Rudolf Ramler, project leader at SCCH. The results are applicable in test management, in predictive assessment of testing effort, and in prioritization and selection of test cases. If a project is under stress and all test cases cannot be conducted as planned; the priorities must be set correctly and first those components must be tested that have a higher probability of defects. Defect prediction provides the basis for this decision.