Meta-analysis has been much debated and criticized since its creation, and it has been argued that systematic reviews are superior. The present study examined the reverse idea – that meta-analysis is the superior method. A literature search was conducted of meta-analysis studies for psychotherapy within the PsycINFO database. The resulting studies were further examined and classified as either primary articles or secondary articles. Primary articles were further examined for secondary articles associated with them. It was found that there was some criticism within the literature. The results were compared to a study of systematic review literature conducted at the same time. The comparison implies that, contrary to the hypothesis, meta-analysis may not be superior to systematic reviews.
A Critical Review of Meta-Analysis
Since its introduction by Gene Glass and his colleagues in 1976, meta-analysis has been a topic of considerable debate (Card, 2011). One of the most common topics of debate is the efficacy of the meta-analytic review, especially when compared to that of the traditional systematic review (Card, 2011). A predominant and noteworthy area of debate and criticism pertaining to the efficacy of meta-analysis is within the field of psychotherapy. In 1977, Glass paired up with Rebecca Smith and released a meta-analytic review of 375 studies on the effectiveness of various psychotherapies (Card, 2011). Their analysis revealed only a small difference amongst the therapies in their level of effectiveness, though their findings have sparked considerable criticism from other psychologists.
Despite this criticism, more and more researchers were beginning to utilize the meta-analytic approach by the early 1980s (Card, 2011), and it continues to be used to this day. The question remains, however, as to which method of analysis is superior: the traditional method of the systematic review or the meta-analytic review. In order to determine this, each review method must be examined in greater detail, with a focus in on meta-analysis.
Both the traditional systematic review and meta-analysis provide a synthesized review of findings from their subject matter, though each goes about this in different ways. While systematic reviews involve analyzing the raw data and results from individually randomized control studies, meta-analysis is a quantitative research method that involves the statistical analysis and comparison of results (effect size) from multiple studies (Glass, 2006). In a book by Larry hedge and Ingram Olkin, the process of meta-analysis is grounded in traditional statistics, providing a sense of formality and statistical value (Card, 2011), as compared to the process and methods of systematic reviews. This use of statistics allows for a comprehensive integration of the study outcomes (Chan & Arvey, 2012). In other words, meta-analysis enables researchers to combine the results from multiple studies, which in turn contributes to the accumulation and development of scientific knowledge. It allows researchers to see what research topics have been explored and the related aspects of that topic, which can aid in the development of future hypotheses or inspire future study ideas (Chan & Arvey, 2012).
Glass (2006) better describes the process behind conducting a meta-analytic review by describing five basic steps involved. The first step is defining the problem, which involves operationally defining any constructs of interest, and constructing a hypothesis. The second step involves selecting any literature for inclusion, based on the relevance of the information to the review and exclusion criteria set out by the reviewer. The third step involves coding the effect size and characteristics of each study involved for categorization purposes. These codes are then transformed into a common scale for statistical analysis, in the...