Industrial Surveys on Software Testing Practices: A Literature Review

Abstract

A US government agency estimated the national cost of inadequate software testing to be \$60 billion annually, and that was 20 years ago. As the role of technology and software has been rapidly increasing worldwide for decades, it suffices to say that the worldwide fiscal effect of poor testing practices today is probably ``quite a bit``.

An increasing number of industry-focused survey studies on testing have been published worldwide in recent years, signalling an increased need to characterize the testing practices of the software development industry. These types of studies can help to guide future research efforts towards subjects that are meaningful to the industry, and provide practitioners with an opportunity to compare their own practice to those of their peers and recognize the main improvement areas.

As no secondary study devoted to these types of survey studies could be identified, the opportunity was seized to carry out a literature review was to find out what the data from these studies can tell us when aggregated. The precise topics focused on were the usage of test levels, test types, test design techniques, test tools and test automation.

Looking at these studies in aggregate tells us about some general trends: unit testing, functional testing and regression testing are popular everywhere, and also quite popular regardless of the surveyed population are performance testing and usability testing. The popularity of the other test levels and test types vary from survey to survey or region to region. Black-box techniques and experience-based techniques are more popular than white-box techniques. Exploratory testing, error guessing, use case testing and boundary value analysis are some of the most popular test design techniques. Much of the industry relies on manual testing over automated testing and/or have inadequately adopted the usage of testing tools.