Software Reliability Engineering: A Roadmap by Michael R. Lyu

Software Reliability Engineering: A Roadmap
Michael R. Lyu

Michael R. Lyu received the Ph.D. in computer science from University of California, Los Angeles in 1988. He is a Professor in the Computer Science and Engineering Department of the Chinese University of Hong Kong. He worked at the Jet Propulsion Laboratory, Bellcore, and Bell Labs; and taught at the University of Iowa. He has participated in more than 30 industrial projects, published over 250 papers, and helped to develop many commercial systems and software tools. Professor Lyu is frequently invited as a keynote or tutorial speaker to conferences and workshops in U.S., Europe, and Asia. He initiated the International Symposium on Software Reliability Engineering (ISSRE) in 1990. He also received Best Paper Awards in ISSRE '98 and in ISSRE '2003. Professor Lyu is an IEEE Fellow and an AAAS Fellow, for his contributions to software reliability engineering and software fault tolerance.

Software Reliability Engineering: A Roadmap
Michael R. Lyu Computer Science and Engineering Department The Chinese University of Hong Kong, Hong Kong [e-mail] various systems and applications has been growing exponentially for the past 40 years [20]. The trend of such growth in the telecommunication, business, defense, and transportation industries shows a compound growth rate of ten times every five years. Because of this ever-increasing dependency, software failures can lead to serious, even fatal, consequences in safety-critical systems as well as in normal business. Previous software failures have impaired several highvisibility programs and have led to loss of business [28]. The ubiquitous software is also invisible, and its invisible nature makes it both beneficial and harmful. From the positive side, systems around us work seamlessly thanks to the smooth and swift execution of software. From the negative side, we often do not know when, where and how software ever has failed, or will fail.

References: [1] S. Amasaki, O. Mizuno, T. Kikuno, and Y. Takagi, “A Bayesian Belief Network for Predicting Residual Faults in Software Products,” Proceedings of 14th International Symposium on Software Reliability Engineering (ISSRE2003), November 2003, pp. 215-226, [2] ANSI/IEEE, Standard Glossary of Software Engineering Terminology, STD-729-1991, ANSI/IEEE, 1991. [3] L. Baresi, E. Nitto, and C. Ghezzi, “Toward Open-World Software: Issues and Challenges,” IEEE Computer, October 2006, pp. 36-43. [4] A. Bertolino, “Software Testing Research: Achievements, Challenges, Dreams,” Future of Software Engineering 2007, L. Briand and A. Wolf (eds.), IEEE-CS Press, 2007. [5] J. Bishop and N. Horspool, “Cross-Platform Development: Software That Lasts,” IEEE Computer, October 2006, pp. 26-35. [6] L. Briand and D. Pfahl, “Using Simulation for Assessing the Real Impact of Test Coverage on Defect Coverage,” IEEE Transactions on Reliability, vol. 49, no. 1, March 2000, pp. 60-70. [7] J. Cheng, D.A. Bell, and W. Liu, “Learning Belief Networks from Data: An Information Theory Based Approach,” Proceedings of the Sixth International Conference on Information and Knowledge Management, Las Vegas, 1997, pp. 325-331. [8] X. Cai and M.R. Lyu, “The Effect of Code Coverage on Fault Detection Under Different Testing Profiles,” ICSE 2005 Workshop on Advances in Model-Based Software Testing (A-MOST), St. Louis, Missouri, May 2005. [9] X. Cai, M.R. Lyu, and K.F. Wong, “A Generic Environment for COTS Testing and Quality Prediction,” Testing Commercial-off-the-shelf Components and Systems, S. Beydeda and V. Gruhn (eds.), Springer-Verlag, Berlin, 2005, pp. 315-347. [10] X. Cai, M.R. Lyu, and M.A. Vouk, “An Experimental Evaluation on Reliability Features of N-Version Programming,” in Proceedings 16th International Symposium on Software Reliability Engineering (ISSRE’2005), Chicago, Illinois, Nov. 8-11, 2005. [11] X. Cai and M.R. Lyu, “An Empirical Study on Reliability and Fault Correlation Models for Diverse Software Systems,” in Proceedings 15th International Symposium on Software Reliability Engineering (ISSRE’2004), Saint-Malo, France, Nov. 2004, pp.125-136. [12] M. Chen, M.R. Lyu, and E. Wong, “Effect of Code Coverage on Software Reliability Measurement,” IEEE Transactions on Reliability, vol. 50, no. 2, June 2001, pp.165-170. [13] M.H. Chen, A.P. Mathur, and V.J. Rego, “Effect of Testing Techniques on Software Reliability Estimates Obtained Using Time Domain Models,” In Proceedings of the 10th Annual Software Reliability Symposium, Denver, Colorado, June 1992, pp. 116-123. [14] J.B. Dugan and M.R. Lyu, “Dependability Modeling for Fault-Tolerant Software and Systems,” in Software Fault Tolerance, M. R. Lyu (ed.), New York: Wiley, 1995, pp. 109–138. [15] D.E. Eckhardt and L.D. Lee, “A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors,” IEEE Transactions on Software Engineering, vol. 11, no. 12, December 1985, pp. 1511–1517. [16] P.G. Frankl and E.J. Weyuker, “An Applicable Family of Data Flow Testing Criteria,” IEEE Transactions on Software Engineering, vol. 14, no. 10, October 1988, pp. 1483-1498. [17] J.R. Horgan, S. London, and M.R. Lyu, “Achieving Software Quality with Testing Coverage Measures,” IEEE Computer, vol. 27, no.9, September 1994, pp. 60-69. [18] C.Y. Huang and M.R. Lyu, “Optimal Release Time for Software Systems Considering Cost, Testing-Effort, and Test Efficiency,” IEEE Transactions on Reliability, vol. 54, no. 4, December 2005, pp. 583-591. [19] C.Y. Huang, M.R. Lyu, and S.Y. Kuo, "A Unified Scheme of Some Non-Homogeneous Poisson Process Models for Software Reliability Estimation," IEEE Transactions on Software Engineering, vol. 29, no. 3, March 2003, pp. 261-269. [20] W.S. Humphrey, “The Future of Software Engineering: I,” Watts New Column, News at SEI, vol. 4, no. 1, March, 2001. [21] V. Issarny, M. Caporuscio, and N. Georgantas: “A Perspective on the Future of Middleware-Based Software Engineering,” Future of Software Engineering 2007, L. Briand and A. Wolf (eds.), IEEE-CS Press, 2007. [22] M. Jazayeri, “Web Application Development: The Coming Trends,” Future of Software Engineering 2007, L. Briand and A. Wolf (eds.), IEEE-CS Press, 2007. [23] Z. Jelinski and P.B. Moranda, “Software Reliability Research,” in Proceedings of the Statistical Methods for the Evaluation of Computer System Performance, Academic Press, 1972, pp. 465-484. [24] B. Littlewood and L. Strigini, “Software Reliability and Dependability: A Roadmap,” in Proceedings of the 22nd International Conference on Software Engineering (ICSE’2000), Limerick, June 2000, pp. 177-188. [25] B. Littlewood and D. Miller, “Conceptual Modeling of Coincident Failures in Multiversion Software,” IEEE Transactions on Software Engineering, vol. 15, no. 12, December 1989, pp. 1596–1614. [26] C. Liu, L. Fei, X. Yan, J. Han, and S. Midkiff, “Statistical Debugging: A Hypothesis Testing-based Approach,” IEEE Transaction on Software Engineering, vol. 32, no. 10, October, 2006, pp. 831-848. [27] N. Looker and J. Xu, “Assessing the Dependability of SOAP-RPC-Based Web Services by Fault Injection,” in Proceedings of 9th IEEE International Workshop on Objectoriented Real-time Dependable Systems, 2003, pp. 163-170. [28] M.R. Lyu (ed.), Handbook of Software Reliability Engineering, IEEE Computer Society Press and McGrawHill, 1996. [29] M.R. Lyu and X. Cai, “Fault-Tolerant Software,” Encyclopedia on Computer Science and Engineering, Benjamin Wah (ed.), Wiley, 2007. [30] M.R. Lyu, Z. Huang, S. Sze, and X. Cai, “An Empirical Study on Testing and Fault Tolerance for Software Reliability Engineering,” in Proceedings 14th IEEE International Symposium on Software Reliability Engineering (ISSRE '2003), Denver, Colorado, November 2003, pp.119-130. [31] Y.K. Malaiya, N. Li, J.M. Bieman, and R. Karcich, “Software Reliability Growth with Test Coverage,” IEEE Transactions on Reliability, vol. 51, no. 4, December 2002, pp. 420-426. [32] T. Margaria and B. Steffen, “Service Engineering: Linking Business and IT,” IEEE Computer, October 2006, pp. 45-55. [33] J.D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper (2nd Edition), AuthorHouse, 2004. [34] J.D. Musa, “Operational Profiles in Software Reliability Engineering,” IEEE Software, Volume 10, Issue 2, March 1993, pp. 14-32. [35] J.D. Musa, A. Iannino, and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGrawHill, Inc., New York, NY, 1987. [36] H. Pham, Software Reliability, Springer, Singapore, 2000. [37] P.T. Popov, L. Strigini, J. May, and S. Kuball, “Estimating Bounds on the Reliability of Diverse Systems,” IEEE Transactions on Software Engineering, vol. 29, no. 4, April 2003, pp. 345–359. [38] S. Rapps and E.J. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Transactions on Software Engineering, vol. 11, no. 4, April 1985, pp. 367375. [39] Rome Laboratory (RL), Methodology for Software Reliability Prediction and Assessment, Technical Report RLTR-92-52, volumes 1 and 2, 1992. [40] M.L. Shooman, Reliability of Computer Systems and Networks: Fault Tolerance, Analysis and Design, Wiley, New York, 2002. [41] R. Taylor and A. van der Hoek, “Software Design and Architecture: The Once and Future Focus of Software Engineering,” Future of Software Engineering 2007, L. Briand and A. Wolf (eds.), IEEE-CS Press, 2007. [42] X. Teng, H. Pham, and D. Jeske, “Reliability Modeling of Hardware and Software Interactions, and Its Applications,” IEEE Transactions on Reliability, vol. 55, no. 4, Dec. 2006, pp. 571-577. [43] L.A. Tomek and K.S. Trivedi, “Analyses Using Stochastic Reward Nets,” in Software Fault Tolerance, M.R. Lyu (ed.), New York: Wiley, 1995, pp. 139–165. [44] W. Torres-Pomales, “Software Fault Tolerance: A Tutorial,” NASA Langley Research Center, Hampton, Virginia, TM-2000-210616, Oct. 2000. [45] K.S. Trivedi, “SHARPE 2002: Symbolic Hierarchical Automated Reliability and Performance Evaluator,” in Proceedings International Conference on Dependable Systems and Networks, 2002. [46] K.S. Trivedi, K. Vaidyanathan, and K. GosevaPostojanova, "Modeling and Analysis of Software Aging and Rejuvenation", in Proceedings of 33 rd Annual Simulation Symposium, IEEE Computer Society Press, Los Alamitos, CA, 2000, pp. 270-279. [47] A. von Mayrhauser and D. Chen, “Effect of Fault Distribution and Execution Patterns on Fault Exposure in Software: A Simulation Study,” Software Testing, Verification & Reliability, vol. 10, no.1, March 2000, pp. 4764. [48] M.A. Vouk, “Using Reliability Models During Testing With Nonoperational Profiles,” in Proceedings of 2nd Bellcore/Purdue Workshop on Issues in Software Reliability Estimation, October 1992, pp. 103-111. [49] W. Wang and M. Tang, “User-Oriented Reliability Modeling for a Web System,” in Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03), Denver, Colorado, November 2003, pp.1-12. [50] M. Xie, Software Reliability Modeling, World Scientific Publishing Company, 1991. [51] S. Yacoub, B. Cukic, and H Ammar, “A Scenario-Based Reliability Analysis Approach for Component-Based Software,” IEEE Transactions on Reliability, vol. 53, no. 4, 2004, pp. 465-480. [52] A.X. Zheng, M.I. Jordan, B. Libit, M. Naik, and A. Aiken, “Statistical Debugging: Simultaneous Identification of Multiple Bugs,” in Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, 2006, pp. 1105-1112.