253 Chapter 6 Evaluation Research Design: Literature Reviews & Synthesis Frequently a research question or hypothesis can be answered through secondary research, i.e., a literature review or synthesis. Both strategies requires the researcher to mine existing data sources; ―pull out‖ relevant data or information; summarize it; logically analyze and/or statistically treat it; and report results. In many instances, the issue, problem, question, etc. which prompted the idea for an evaluation study is resolved or answered by studying and reporting previous research reports in the literature or synthesizing, either logically, statistically, or both, data drawn from existing databases. In program evaluation, it is often necessary to describe or express the rationale for how a program is/was intended to function to achieve its goals and objectives. Such a rationale is called the program’s theory. Program theory is integral to explaining why a program ―worked‖ or ―didn't work,‖ drawing conclusions about the program's implementation or impact, and framing recommendations for decision-making. Evaluation research proposals serve as the basis for negotiating a study commission, project contract, and conducting the actual evaluation research study. Once a study has been completed, it is necessary to file a written evaluation report. Presented in this chapter are a comparison of the traditional literature review, metaanalysis, and best evidence synthesis and a description of (1) how to conduct and report a literature review or syntheses; (2) express a program’s theory; and (3) format an evaluation research proposal and report. Presented in chapter appendices are (1) social science and business data sources, (2) a tool for assessing and evaluating research articles, (3) budgeting basics for use in preparing an evaluation research proposal and report, and (4) a sample graduate task description for preparing a program evaluation research proposal. I. Comparing the Traditional Literature Review, Meta-Analysis, and Best Evidence Synthesis A. Suri (2000) explains that a single study is usually unable to generate generalizable results or definitively answer issues under investigation in the socio-behavioral sciences. Based on this assumption, Suri goes on to critique three contemporary methods of literature synthesis: traditional narrative reviews of research, metaanalysis, and best evidence synthesis. B. Suri (2000) identified the primary strength of the traditional narrative review of research as methodological flexibility when the review is undertaken by a highly experienced research reviewer. However, the principle weaknesses of this approach include reviewer subjectivity, unclear article inclusion criteria, different methodologies employed by the empirical research studies included in the review, inconclusive results as evidenced by conflicting conclusions and/or hypotheses. It is these probable weaknesses, coupled with research reviewer subjectivity, which may contribute to the failure to find any effects, inconsistent effects, or false effects upon the problem or issue under investigation. This writer would also add
254 that frequently, authors of traditional literature reviews are not necessarily highly experienced research reviewers and as a consequence such traditional reviews are often characterized by uneven levels of quality and accuracy of interpreting and distilling the results of studies included in the review. Suri argues that in analyzing any relevant empirical literature, it is critical to identify and analyze variables which may mediate the effects of the independent variable(s) on the dependent variable; Suri argues that traditional narrative research reviews frequently fail to identify, let alone explain these variables and their possible effect on the dependent variable. Confounding variables may limit, block, or enhance any effect an independent variable may have on the dependent variable; thereby, leading to false,...
Please join StudyMode to read the full document