Evaluation Systems and Systems Evaluation:
Building Capacity and Tools for Enhancing STEM Education Evaluation This research is designed to enhance the capability of the field of evaluation to develop more effective evaluation systems and to conduct more rigorous systems evaluation. By evaluation system we mean the comprehensive and integrated set of capabilities, resources, activities and support mechanisms for conducting evaluation work. By systems evaluation, we mean the assessment of the functions, products, outcomes and impacts of a set of programs, activities or interventions. The intellectual merit of this research stems from its contribution to the improvement of our understanding of the role of systems in evaluation specifically through (1) development of a formal model of evaluation system needs that provides structure for evaluation capacity building and delivery; (2) development of one or more measures of evaluation systems capacity; and (3) conducting a pilot test of an evaluation capacity-building system and measure(s). The broader impacts of this research stem from the way it will: strengthen the infrastructure for research and evaluation through the development of new measures for local evaluation capacity building ; and, develop a new evaluation network that will serve as a model for multi-institutional multi-program evaluation partnerships.
The specific context for this effort is the evaluation of Science, Technology, Engineering and Mathematics (STEM) education programs, but its implications extend to the field of evaluation generally. The research will result in a set of specific products that will have immediate utility in STEM education and be adaptable to non-STEM contexts, including a measurement approach and instrument(s) that can be used to assess and monitor evaluation capacity and support in a STEM outreach and educational system. This project is directed at how we might develop more effective evaluation systems both generally and for STEM education programs in particular. The landscape of contemporary evaluation in recent years is replete with evaluation systems development efforts, yet we know surprisingly little about how such endeavors might best be developed and themselves evaluated. This project uses structured participatory mixed methods systems approaches to: identify the needs that STEM educators and evaluators have for evaluation of their programs; develop a formal model of an evaluation system to address those needs; identify the major components of that system and how they might work together; and, operationalize one or more measures of evaluation capacity building that can be used to assess the development of that evaluation system. The model and measures will be implemented and tested in a pilot project in New York City that includes both formal and informal STEM education programs. The project will contribute to our fundamental knowledge regarding the nature and structure of evaluation systems, how they can be developed more effectively and how to evaluate their development. While this particular project involves STEM education and outreach specifically, the completed models and measures for evaluation systems will be useful in many substantive areas that extend far beyond the STEM context. Thus, this project generates new knowledge in the evaluation of STEM education specifically and explores new concepts in the evaluation field generally. In addition, the model and measures proposed here will enhance the instrumentation, networks, and partnerships developed to support both STEM evaluation and systems of evaluation.
The research project proposed here, is related to the idea of systems and systems thinking. It will contribute to our understanding of how evaluation systems that are regional and include university, school and community components might best be developed and integrated, and will provide specific measures and tools of relevance for monitoring...