A Case Study in Successful Risk-Based Testing at CA Introduction This article presents a case study of a risk-based testing pilot project at CA, the world's leading independent IT management software company. The development team chosen for this pilot is responsible for a widely-used mainframe software product called CA SYSVIEW® Performance Management, an intuitive tool for proactive management and real-time monitoring of z/OS environments. By analyzing a vast array of performance metrics, CA SYSVIEW can help organizations identify and resolve problems quickly. Companies are highly dependent on the reliability of their mainframe systems. If the mainframe doesn’t run, the company stops. Mainframe workloads also are growing considerably as companies’ businesses grow and as they continually seek to leverage data and applications in new ways. At the same time, these companies are losing their experienced mainframe workforce, largely to retirement. This makes the quality of their mainframe management tools even more important to them. CA piloted risk-based testing as part of our larger effort to ensure the quality of the solutions we deliver. The pilot consisted of six main activities: • • • • • • Training key stakeholders on risk-based testing Holding a quality risk analysis session Analyzing and refining the quality risk analysis Aligning the testing with the quality risks Guiding the testing based on risks Assessing benefits and lessons
This article addresses each of these areas – as well as some of the broader issues associated with risk-based testing. What is Risk-Based Testing? Generally speaking, risk is the possibility of a negative or undesirable outcome or event. Testing is concerned with two main types of risks: • Product or quality risks, which are problems that can potentially affect the quality of the product itself, such as a defect that could cause a system to crash during normal operation. Project or planning risks, which are problems that can potentially affect overall project success, such as a staffing shortage that could delay completion of a deliverable.
Of course, not all risks are equal and there are a number of ways to classify the different levels of risk. The simplest is to look at two factors:
Rex Black, Ken Young, and Peter Nash
Page 1 of 11
A Case Study in Successful Risk-Based Testing at CA • The likelihood of the problem occurring, which depends primarily on technical considerations, such as the programming languages used and the constraints of a given computing platform. The impact of the problem should it occur, which depends primarily on business considerations, such as the financial impact of system downtime or the amount of lost staff productivity.
Risk-based testing is guided by the level of risk associated with items identified during analysis. Although risk can guide testing in various ways, there are three common ones. First, during all test activities, test teams allocate effort to each quality risk item based on the relative level of risk. Test managers and analysts align the rigor and extensiveness of test techniques with the level of risk. They carry out test activities in risk order, starting with the most important risks. They also work with the project team to prioritize resolution of discovered defects based on the level of risk. Second, test managers implement control steps for all significant identified project risks. A control step is either a mitigation (something done in advance to reduce the likelihood and/or impact of a risk) or a contingency (something you are prepared to do if the risk becomes an event to reduce the impact of the event). The higher the level of risk, the more thoroughly that project risk is controlled. These project risks must include risks related to testing itself, since problems during test execution can reduce test scope and thereby result in quality risks. Third, test managers and test analysts report...
Please join StudyMode to read the full document