Agile

ARTICLE IN PRESS
Available online at www.sciencedirect.com

Information and Software Technology xxx (2008) xxx–xxx www.elsevier.com/locate/infsof

Empirical studies of agile software development: A systematic review
˚ Tore Dyba *, Torgeir Dingsøyr
SINTEF ICT, S.P. Andersensv. 15B, NO-7465 Trondheim, Norway Received 22 October 2007; received in revised form 22 January 2008; accepted 24 January 2008

Abstract Agile software development represents a major departure from traditional, plan-based approaches to software engineering. A systematic review of empirical studies of agile software development up to and including 2005 was conducted. The search strategy identified 1996 studies, of which 36 were identified as empirical studies. The studies were grouped into four themes: introduction and adoption, human and social factors, perceptions on agile methods, and comparative studies. The review investigates what is currently known about the benefits and limitations of, and the strength of evidence for, agile methods. Implications for research and practice are presented. The main implication for research is a need for more and better empirical studies of agile software development within a common research agenda. For the industrial readership, the review provides a map of findings, according to topic, that can be compared for relevance to their own settings and situations. Ó 2008 Elsevier B.V. All rights reserved.
Keywords: Empirical software engineering; Evidence-based software engineering; Systematic review; Research synthesis; Agile software development; XP; Extreme programming; Scrum

Contents 1. 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Background – agile software development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1. The field of agile software

References: [1] North American and European Enterprise Software and Services Survey, Business Technographics Ed., 2005. [2] P. Abrahamsson, O. Salo, J. Ronkainen, J. Warsta, Agile software development methods: review and analysis, VTT Technical report, 2002. [3] P. Abrahamsson, J. Warsta, M.T. Siponen, J. Ronkainen, New directions on agile methods: a comparative analysis, in: Proceedings of the 25th International Conference on Software Engineering (ICSE’03), IEEE Press, 2003. [4] R.L. Ackoff, Alternative types of planning, in: Ackoff’s Best: His Classic Writings on Management, Wiley, New York, 1999, pp. 104– 114. [5] A. Anderson, R. Beattie, K. Beck, D. Bryant, M. Dearment, M. Fowler, M. Fronczak, R. Garzaniti, D. Gore, B. Hacker, C. Hendrickson, R. Jeffries, D. Joppie, D. Kim, P. Kowalsky, D. Mueller, T. Murasky, R. Nutter, A. Pantea, D. Thomas, Chrysler goes to extremes, Distributed Computing Magazine (October) (1998) 24–28. [6] M. Aoyama, Web-based agile software development, IEEE Software 15 (6) (1998) 56–65. [7] D. Atkins, D. Best, P.A. Briss, M. Eccles, Y. Falck-Ytter, S. Flottorp, G.H. Guyatt, R.T. Harbour, M.C. Haugh, D. Henry, S. Hill, R. Jaeschke, G. Leng, A. Liberati, N. Magrini, J. Mason, P. Middleton, J. Mrukowicz, D. O’connell, A. D Oxman, B. Phillips, H.J. Schunemann, T.T.-T. Edejer, H. Varonen, G.E. Vist, J.W. Williams ¨ Jr., Z. Stephanie, Grading quality of evidence and strength of recommendations, BMJ 328 (1490) (2004). [8] D. Avison, F. Lau, M. Myers, P.A. Nielsen, Action research, Communications of the ACM 42 (1) (1999) 94–97. [9] K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, 2000, ISBN 0-201-61641-6. [10] K. Beck, Extreme Programming Explained: Embrace Chage, second ed., Addison-Wesley, 2004, ISBN 978-0321278654. [11] B. Boehm, Get ready for agile methods, with care, IEEE Computer 35 (1) (2002) 64–69. [12] B. Boehm, R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley, Boston, 2003, ISBN 978-0321186126. [13] N. Britten, R. Campbell, C. Pope, J. Donovan, M. Morgan, R. Pill, Using meta ethnography to synthesise qualitative research: a worked example, Journal of Health Services Research and Policy 7 (4) (2002) 209–215. [14] P. Checkland, J. Scholes, Soft Systems Methodology in Action, Wiley, Chichester, 1990, ISBN 0-471-98605-4. [15] A. Cockburn, Selecting a project’s methodology, IEEE Software 17 (4) (2000) 64–71. [16] A. Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams, Addison-Wesley, 2004, ISBN 0-201-69947-8. [17] D. Cohen, M. Lindvall, P. Costa, An introduction to agile methods, in: M.V. Zelkowitz (Ed.), Advances in Computers, Advances in Software Engineering, vol. 62, Elsevier, Amsterdam, 2004. [18] J. Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20 (1960) 37–46. [19] K. Conboy, B. Fitzgerald, Toward a conceptual framework of agile methods: a study of agility in different disciplines, in: Proceedings of XP/Agile Universe, Springer Verlag, 2004. [20] Y. Dittrich, M. John, J. Singer, B. Tessem, For the special issue on qualitative software engineering research, Information and Software Technology 49 (6) (2007) 531–539. ˚ [21] T. Dyba, Improvisation in small software organizations, IEEE Software 17 (5) (2000) 82–87. ˚ [22] T. Dyba, E. Arisholm, D. Sjøberg, J. Hannay, F. Shull, Are two heads better than one? On the effectiveness of pair-programming, IEEE Software 24 (6) (2007) 10–13. ˚ [23] T. Dyba, T. Dingsøyr, G.K. Hanssen, Applying systematic reviews to diverse study types: an experience report, in: Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07), IEEE Computer Society, Madrid, Spain, 2007, pp. 225–234. ˚ [24] T. Dyba, B. Kitchenham, M. Jørgensen, Evidence-based software engineering for practitioners, IEEE Software 22 (1) (2005) 58–65. [25] A.C. Edmondson, S.E. Mcmanus, Methodological fit in management field research, Academy of Management Review 32 (4) (2007) 1155– 1179. [26] H. Erdogmus, M. Morisio, M. Torchiano, On the effectiveness of the test-first approach to programming, IEEE Transactions on Software Engineering 31 (3) (2005) 226–237. [27] J. Erickson, K. Lyytinen, K. Siau, Agile Modeling, Agile software development, and extreme programming: the state of research, Journal of Database Management 16 (4) (2005) 88–100. [28] T. Gilb, Competitive Engineering: A Handbook for Systems Engineering, Requirements Engineering, and Software, Elsevier Butterworth-Heinemann, Oxford, 2005, ISBN 0-7506-6507-6. [29] T. Greenhalgh, How to Read a Paper, second ed., BMJ Publishing Group, London, 2001. [30] A. Gunasekaran, Agile manufacturing: A framework for research and development, International Journal of Production Economics 62 (1– 2) (1999) 87–105. ˚ Please cite this article in press as: T. Dyba, T. Dingsøyr, Empirical studies of agile software development: A systematic review, Inform. Softw. Technol. (2008), doi:10.1016/j.infsof.2008.01.006 ARTICLE IN PRESS ˚ T. Dyba, T. Dingsøyr / Information and Software Technology xxx (2008) xxx–xxx [31] J.P.T. Higgins, S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.0 (updated February 2008), The Cochrane Collaboration, 2008. Available from: . [32] W.S. Humphrey, PSP: A Self-Improvement Process for Software Engineers, Addison-Wesley, 2005, ISBN 978-0321305497. [33] M. Host, P. Runeson, Checklists for software engineering case study ¨ research, in: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, IEEE, Madrid, Spain, 2007, pp. 479–481. [34] G. Keefer, Extreme Programming Considered Harmful for Reliable Software Development 2.0, AVOCA GmbH, Online Report, 2003. [35] K.S. Khan, G. Ter Riet, J. Glanville, A.J. Sowden, J. Kleijnen, Undertaking Systematic Review of Research on Effectiveness, CRD’s Guidance for those Carrying Out or Commissioning Reviews, CRD Report Number 4, second ed., NHS Centre for Reviews and Dissemination, University of York, 2001. [36] B.A. Kitchenham, Guidelines for performing Systematic Literature Reviews in Software Engineering Version 2.3, Keele University and University of Durham, EBSE Technical Report, 2007. [37] B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, J. Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering 28 (8) (2002) 721–734. [38] P. Krutchen, The Rational Unified Process: An Introduction, third ed., Addison-Wesley, Boston, 2003. [39] J.R. Landis, G.G. Koch, The measurement of observer agreement for categorical data, Biometrics 33 (1) (1977) 159–174. [40] C. Larman, V.R. Basili, Iterative and incremental development: a brief history, IEEE Computer 36 (6) (2003) 47–56. [41] J. Mcavoy, T. Butler, The impact of the Abilene Paradox on doubleloop learning in an agile team, Information and Software Technology 49 (6) (2007) 552–563. [42] P. Mcbreen, Questioning Extreme Programming, Pearson Education, Boston, MA, USA, 2003, ISBN 0-201-84457-5. [43] H. Merisalo-Rantanen, T. Tuure, R. Matti, Is extreme programming just old wine in new bottles: a comparison of two cases, Journal of Database Management 16 (4) (2005) 41–61. [44] P. Meso, R. Jain, Agile software development: adaptive systems principles and best practices, Information Systems Management 23 (3) (2006) 19–30. [45] M.B. Miles, M. Huberman, Qualitative Data Analysis: An Expanded Sourcebook, second ed., Sage Publications, 1994, ISBN 0803955405. [46] S. Nerur, V. Balijepally, Theoretical reflections on agile development methodologies, Communications of the ACM 50 (3) (2007) 79–83. [47] S. Nerur, R. Mahapatra, G. Mangalaraj, Challenges of migrating to agile methodologies, Communications of the ACM (May) (2005) 72– 78. [48] G.W. Noblit, R.D. Hare, Meta-Ethnography: Synthesizing Qualitative Studies, Sage Publications, London, 1988. [49] T. Ohno, Toyota Production System: Beyond Large-scale Production, Productivity Press, New York, USA, 1988, ISBN 0-915299-14-3. 27 [50] S.R. Palmer, J.M. Felsing, A Practical Guide to Feature-driven Development, Prentice Hall, Upper Saddle River, NJ, 2002, ISBN 013-067615-2. [51] M.C. Paulk, C.V. Weber, B. Curtis, M.B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, 1995, ISBN 0-201-54664-7. [52] M. Poppendieck, T. Poppendieck, Lean Software Development – An Agile Toolkit for Software Development Managers, Addison-Wesley, Boston, 2003, ISBN 0-321-15078-3. [53] V. Rajlich, Changing the paradigm of software engineering, Communications of the ACM 49 (8) (2006) 67–70. [54] C.K. Riemenschneider, B.C. Hardgrave, F.D. Davis, Explaining software developer acceptance of methodologies: a comparison of five theoretical models, IEEE Transactions on Software Engineering 28 (12) (2002) 1135–1145. [55] L.M. Sanchez, R. Nagi, A review of agile manufacturing systems, International Journal of Production Research 39 (16) (2001) 3561– 3600. [56] K. Schwaber, M. Beedle, Agile Software Development with Scrum, Prentice Hall, Upper Saddle River, 2001. [57] W.R. Shadish, T.D. Cook, D.T. Campbell, Experimental and QuasiExperimental Designs for Generalized Causal Inference, Houghton Mifflin Company, Boston, 2002. ˚ [58] D. Sjøberg, T. Dyba, M. Jørgensen, The Future of Empirical Methods in Software Engineering Research, in: Future of Software Engineering (FOSE’07), IEEE, 2007, pp. 358–378. [59] D. Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.-K. Liborg, A.C. Rekdal, A survey of controlled experiments in software engineering, IEEE Transactions on Software Engineering 31 (9) (2005) 733–753. [60] J. Stapleton, DSDM: Business Focused Development, second ed., Pearson Education, 2003, ISBN 978-0321112248. [61] M. Stephens, D. Rosenberg, Extreme Programming Refactored: The Case Against XP, Apress, Berkeley, CA, 2003, ISBN 1-59059-096-1. [62] A. Strauss, J. Corbin, Basics of Qualitative Research, second ed., Sage Publications, 1998, ISBN 0-8039-5939-7. [63] H. Takeuchi, I. Nonaka, The new product development game, Harvard Business Review (January) (1986) 137–146. [64] D. Turk, R. France, B. Rumpe, Assumptions underlying agile softwaredevelopment processes, Journal of Database Management 16 (4) (2005) 62–87. [65] E. Wenger, Communities of Practice: Learning, Meaning and Identity, Cambridge University Press, Cambridge, UK, 1998, ISBN 0-52143017-8. [66] L. Williams, A. Cockburn, Agile software development: it’s about feedback and change, IEEE Computer 36 (6) (2003) 39–43. [67] J.P. Womack, D.T. Jones, D. Roos, The Machine that Changed the World: The Story of Lean Production – Toyota’s Secret Weapon in the Global Car Wars that is Now Revolutionizing World Industry, Free Press, 2007, ISBN 978-0743299794. ˚ [68] P. Agerfalk, B. Fitzgerald, Flexible and distributed software processes: old petunias in new bowls? Communications of the ACM 49 (10) (2006) 27–34. ˚ Please cite this article in press as: T. Dyba, T. Dingsøyr, Empirical studies of agile software development: A systematic review, Inform. Softw. Technol. (2008), doi:10.1016/j.infsof.2008.01.006