Chapter 22 Meta-Analysis
‘Fett’s Law: Never replicate a successful experiment’
Content list What is meta-analysis? Examples of meta-analytic studies Conducting a meta-analysis Replication and meta-analysis Comparing studies by effect size Combining studies by effect size Comparing studies by signiﬁcance levels Combining studies by signiﬁcance levels Comparing and combining more than two effect sizes and signiﬁcance levels Some issues in meta-analysis 533 535 536 539 540 542 544 545 547 547
By the end of this chapter you will understand:
1 What meta-analysis is. 2 How it helps in conﬁrming research ﬁndings. 3 How to undertake a meta-analysis.
Introduction Meta-analysis has become an important research strategy as it enables researchers to combine the results of many pieces of research on a topic to determine whether the ﬁnding holds generally. This is better than trying to assume that the ﬁndings of a single study have global meaning.
What is meta-analysis?
Each strand of a rope contributes to the strength of the rope. But the rope is stronger than any individual strand. Similarly, when a particular ﬁnding is obtained repeatedly,
EXTENSION CHAPTERS ON ADVANCED TECHNIQUES
under a variety of conditions, we are strongly conﬁdent that there exists a general principle. The results of small localized individual studies, no matter how well conducted, are often insufﬁcient to provide us with conﬁdent answers to questions of general importance. Meta-analysis allows us to compare or combine results across a set of similar studies. In the individual study, the units of analysis are the individual observations. In meta-analysis the units of analysis are the results of individual studies. The term meta-analysis means ‘an analysis of analysis’. A particular topic may have been replicated in various ways, using, for example, differently sized samples, and conducted in different countries under different environmental, social and economic conditions. Sometimes results appear to be reasonably consistent; others less so. Meta-analysis enables a rigorous comparison to be made rather than a subjective ‘eyeballing’. However, the technique relies on all relevant information being available for each of the examined studies. If some crucial factors like sample size and methodology are missing then comparison is not feasible.
Meta-analysis. An objective and quantitative methodology for synthesizing previous studies and research on a particular topic into an overall ﬁnding.
If you are asked to write a report for your manager on the economics of recycling waste paper, or on the relationship between air travel fares and ticket sales, or on the relationship between money supply and mortgage rates, you will search for information from a variety of sources, including in-house documents, the Internet, the national and local libraries, etc. The strategy is to read studies relevant to the topic you wish to investigate, summarize the ﬁndings, and then integrate the existing knowledge. From this you may conclude that a particular variable is of crucial importance, or that the relationships between particular variables are worthy of note. This is the standard literature survey where you draw essentially subjective conclusions based on your critical evaluation of the literature. You often use a ‘voting method’ as a crude index of where the balance of results lies. This method is ﬂawed and inexact because: • • • you are unable to deal with the large number of studies on a topic, and so focus on a small subset of studies, often without describing how the subset was selected; you often cite the conclusions of previous reviews without examining those reviews critically; you are interested in a particular issue so you might not be inclined to give full weight to evidence that is contrary to your own desired outcome.
As a result, your subjective conclusion may not accurately reﬂect the actual strength of the relationship. You...
References: Cooper, H. & Rosenthal, R. 1982. Statistical versus traditional methods for summarising research ﬁndings. Psychological Bulletin, 87, 442–449. De Dreu, C.K. & Weingart, L.R. 2003. Task versus relationship conﬂict, team performance and team member satisfaction: a meta analysis. Journal of Applied Psychology, 88 (4), 741–749. Gully, S., Incalcaterra, K., Joshi, A. & Beaubien, J. 2002. A meta analysis of team efﬁciency, potency and performance. Journal of Applied Psychology, 87 (5), 819–832. Hosada, M., Stone-Romero, E. & Coats, G. 2003. The effects of physical attractiveness on job related outcomes. Personnel Psychology, 56 (2), 431–448. Iaffaldano, M. & Muchinsky, P.M. 1985. Job satisfaction and job performance: a meta analysis. Psychological Bulletin, 97, 251–273. Jenkins, J. 1986. Financial incentives. In Generalising from Laboratory to Field Settings. Locke, E. (ed). Lexington: Lexington Books. Judge, T.A., Colberet, A. & Illies, R. 2004. Intelligence and leadership. Journal of Applied Psychology, 89 (3), 542–552. Mullen, B. & Copper, C. 1994. The relation between group cohesion and performance. Psychological Bulletin, 115 (2), 210–227. Mullen, B. & Rosenthal, R. 1985. Basic Meta-analysis. Hillsdale: Lawrence Erlbaum. Rosenthal, R. 1979. The ﬁle drawer problem. Psychological Bulletin, 86, 638–641. Rosenthal, R. 1984. Meta analytic procedures for social research, in Applied Social Science Research Methods, Vol 6. Beverly Hills: Sage. Rosenthal, R. 1994. Interpersonal expectancy effects. A 30 year perspective. Current Directions in Psychological Science, 3, 176–179. Rosenthal, R. 1991. Meta-analytic Procedures for Social Research. Newbury Park: Sage. Smith, M. & Glass, G. 1977. Meta-analysis of psychotherapy outcome studies. American Psychologist. 32, 752–760. Thorsteinson, T.J. 2003. Job attitudes of part-time vs. full-time workers. Journal of Occupational and Organisational Psychology, 76 (2), 151–177.
You will never have to undertake meta-analysis by hand. There are computer programs for this. For example: 1 Meta-Stat – A Tool for the Meta-Analysis of Research Studies. Produced by Lawrence M. Rudner, Gene V. Glass, David L. Evartt and Patrick J. Emery. Meta-Stat is a DOSbased computer program that automates the many complex tasks that are required to
perform a meta-analysis. The data can easily be output in a format ready for use by SPSS. Meta-Stat is free for non-commercial, educational use. Meta-Stat is available through the auspices of the ERIC Clearinghouse on Assessment and Evaluation, Department of Measurement, Statistics and Evaluation, University of Maryland, College Park. Comprehensive Meta-Analysis Program (CMA) produced by Biostat. A free trial download is available at www.power.analysis.com/about.biostat.htm
Now turn to the website page for this chapter and undertake the activities there.
Please join StudyMode to read the full document