Douglas Hoffman, BACS, MSEE, MBA, CSQE Software Quality Methods, LLC. 24646 Heather Heights Place Saratoga, California 95070-9710 Phone 408-741-4830 Fax 408-867-4550 firstname.lastname@example.org
Keywords: Automated Testing, Automation Tools, Cost of Testing, Intangible Costs, Return on Investment, Tangible Costs
Many managers today expect software test automation to be a silver bullet; killing the problems of test scheduling, the costs of testing, defect reporting, and more. Automating testing can have positive impacts in many areas, and there are many success stories to provide hope that test automation will save money and solve some testing problems. Unfortunately, there are many more horror stories, disappointments, and bad feelings, even in cases where automation has been beneficial. I have been brought into more than one situation where previous attempts at automating software testing have failed; where large investments have been made in shelfware, and many years of effort creating automated tests abandoned. The purpose of this paper is to provide some practical guidance for understanding and computing cost and benefits from test automation. It describes some financial, organizational, and test effectiveness impacts observed when software test automation is installed. The paper also advises about areas that are difficult or impossible to factor into the financial equations and addresses some common misconceptions management holds about test automation. There are many factors to consider when planning for software test automation. Automation changes the complexion of testing and the test organization from design through implementation and test execution. It usually has broad impacts on the organization in such things as the tasks performed, test approaches, and even product features. There are tangible and intangible elements and widely held myths about benefits and capabilities of test automation. It is important to really understand the potential costs and benefits before undertaking the kind of change automation implies if only to plan well to make the most of them. Organizational impacts include such things as the skills needed to design and implement automated tests, automation tools, and automation environments. Development and maintenance of automated tests is quite different from manual tests. The job skills change, test approaches change, and testing itself changes when automation is installed. Automation has the potential for changing the product being tested and the processes used for development and release. These impacts have positive and negative components that must be considered.
© 1999, Software Quality Methods, LLC.
Page 1 of 14
Cost Benefits Analysis of Test Automation
Setting realistic expectations in management and understanding where benefits should be derived from test automation are key to success. We can easily provide cost justification for proposed automation if management demands numbers. Tool vendors and experts publishing their test automation strategies provide excellent sources of equations and customer examples justifying almost any approach. In my experience the trick has been to figure out what costs and benefits really relate to the automation at hand, and how to make best use of them. It is critical to keep in mind that the purpose of test automation is to do testing better in some way. Automation is only a means to help accomplish our task – testing a product. The automation itself generally does not otherwise benefit the organization any more than the testing does. Cost benefit analysis provides us with useful information for deciding how to best manage and invest in testing. There are also many automation areas that have the potential to provide a benefit or be a drawback depending on how they are handled. For example, automated tests may reduce staff involvement during testing, thus saving in...