Evaluation Systems and Systems Evaluation

C. PROJECT DESCRIPTION
Evaluation Systems and Systems Evaluation:
Building Capacity and Tools for Enhancing STEM Education Evaluation
This research is designed to enhance the capability of the field of evaluation to develop more effective evaluation systems and to conduct more rigorous systems evaluation. By evaluation system we mean the comprehensive and integrated set of capabilities, resources, activities and support mechanisms for conducting evaluation work. By systems evaluation, we mean the assessment of the functions, products, outcomes and impacts of a set of programs, activities or interventions. The intellectual merit of this research stems from its contribution to the improvement of our understanding of the role of systems in evaluation specifically through (1) development of a formal model of evaluation system needs that provides structure for evaluation capacity building and delivery; (2) development of one or more measures of evaluation systems capacity; and (3) conducting a pilot test of an evaluation capacity-building system and measure(s). The broader impacts of this research stem from the way it will: strengthen the infrastructure for research and evaluation through the development of new measures for local evaluation capacity building ; and, develop a new evaluation network that will serve as a model for multi-institutional multi-program evaluation partnerships.
The specific context for this effort is the evaluation of Science, Technology, Engineering and Mathematics
(STEM) education programs, but its implications extend to the field of evaluation generally. The research will result in a set of specific products that will have immediate utility in STEM education and be adaptable to non-STEM contexts, including a measurement approach and instrument(s) that can be used to assess and monitor evaluation capacity and support in a STEM outreach and educational system.
This project is directed at how we might develop more effective

References: Anderberg, M. R. (1973). Cluster analysis for applications. New York, NY: Academic Press, Inc. Artelt, T. A. (2001). Evidence-based social work practice with families: A lifespan approach. Research on Social Work Practice, 11(3), 404-405. Baron, J. (2002). Bringing evidence-driven progress to education: A recommended strategy for the u.S. Bertalanffy, L. V. (1995). General system theory: Foundations, development, applications (Rev. ed. ed.). Biegel, D. E., Johnsen, J. A., & Shafran, R. (1997). Overcoming barriers faced by african-american families with a family member with mental illness Brainard, J. (2002). White house proposes criteria for measuring progress and results of federally financed research Briss, P. A., Zaza, S., Marguerite, P., Fielding, J., Wright-De Aguero, L., Truman, B. I., et al. (2000). Brownson, R. C., Baker, E. A., Leet, T. L., & Gillespie, K. N. (Eds.). (2002). Evidence-based public health: Oxford University Press. Chalmers, I., & Haynes, B. (1994). Systematic reviews - reporting, updating, and correcting systematic reviews of the effects of health-care Chen, H. T., & Rossi, P. H. (1990). Theory-driven evaluations. Thousand Oaks, CA: Sage. Compton, D. W., Baizerman, M., & Stockdill, S. H. (Eds.). (2002). The art, craft, and science of evaluation capacity building (Vol Coxon, A. P. M. (1999). Sorting data: Collection and analysis. Thousand Oaks, CA: Sage. Daughtry, D., & Kunkel, M. A. (1993). Experience of depression in college-students - a concept map. Davies, P. (1999). What is evidence-based education? British Journal of Educational Studies, 47(2), 108121. Davison, M. L. (1983). Multidimensional scaling. New York, NY: John Wiley & Sons, Inc. DeRidder, D., Depla, M., Severens, P., & Malsch, M. (1997). Beliefs on coping with illness: A consumer 's perspective Donnelly, K. Z., Donnelly, J. P., & Grohman, K. J. (2000). Cognitive, emotional, and behavioral problems associated with traumatic brain injury: A concept map of patient, family, and provider Dopson, S., Locock, L., Gabbay, J., Ferlie, E., & Fitzgerald, L. (2003). Evidence-based medicine and the implementation gap Everitt, B. (1980). Cluster analysis (second edition) "a review of clustering technique". New York, NY: Halsted Press, Division of John Wiley & Sons, Inc. Evidence-Based Medicine Working Group. (1992). Evidence-based medicine: A new approach to teaching the practice fo medicine Fetterman, D., Kaftarian, S. J., & Wandersman, A. (1996). Empowerment evaluation: Knowledge and tools for self-assessment and accountability Frechtling, J. (2002). The 2002 user friendly handbook for project evaluation. Alexandria, VA.: The National Science Foundation. Frechtling, J., & Sharp, L. (Eds.). (1997). User-friendly handbook for mixed-methods evaluations. Gibbs, L. E. (2003). Evidence-based practice for the helping professions. Pacific Grove, CA: Thomson, Brooks-Cole. Hausman, A. J. (2002). Implications of evidence-based practice for community health. American Journal of Community Psychology, 30(3), 453-467. Holland, J. (1995). Hidden order: How adaptation builds complexity. Reading, MA: Perseus Books. Holland, J. H. (1998). Emergence: From chaos to order. Reading, Mass.: Addison-Wesley. Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Johnsen, J. A., Biegel, D. E., & Shafran, R. (2000). Concept mapping in mental health: Uses and adaptations Juni, P., Witschi, A., Bloch, R., & Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta- analysis Kellogg Foundation. (2001). Logic model development guide: Using logic models to bring together planning, evaluation and action Krippendorf, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Newbury Park, CA: Sage Publications. Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage Publications. Lipsey, M., & Wislson, D. B. (2000). Practical meta-analysis. Newbury Park, CA: Sage. Macaulay, A. C. (1999). Participatory research maximizes community and lay involvement. British Medical Journal, 319(7212), 774-778. O 'Fallon, L. R., & Dearry, A. (2002). Community-based participatory research as a tool to advance environmental health sciences Pappaioanou, M., & Evans, C. A. (1998). Development of the guide to community preventive services: A u.S Peters, S. J., Jordan, N. R., Adamek, M., & Alter, T. R. (Eds.). (2005). Engaging campus and community: The practice of public scholarship in the american state and land-grant university system Peters, S. J., Jordan, N. R., Alter, T. R., & Bridger, J. C. (2003). The craft of public scholarship in landgrant education. Journal of Higher Education Outreach and Engagement, 8(1), 75-86. Pirrie, A. (2001). Evidence-based practice in education: The best medicine? British Journal of Educational Studies, 49(2), 124-136. Reason, P., & Bradbury, H. (Eds.). (2001). Handbook of action research: Participative inquiry and practice Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage. Sackett, D. L. (1997). Evidence-based medicine. Semin Perinatol, 21(1), 3-5. Sackett, D. L., Richardson, S., Rosenberg, W., & Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach ebm Shadish, W. R. (1986). Planned critical multiplism - some elaborations. Behavioral Assessment, 8(1), 75103. Shern, D., Trochim, W., & Christina, L. (1995). The use of concept mapping for assessing fidelity of model transfer: An example from psychiatric rehabilitation (2) p. 1442-153. Skelcher, C. (2002). What works? Evidence-based policy and practice in public services. Local Government Studies, 28(2), 105-106. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752-760. Southern, D. M., Young, D., Dunt, D., Appleby, N. J., & Batterham, R. W. (2002). Integration of primary health care services: Perceptions of australian general practitioners, non-general practitioner health