Eğitim Programları ve Öğretim Kongresi (EPOK), İzmir, Turkey, 4 - 06 November 2021
This study aims to conduct a meta-evaluation of the program evaluation research titled Evaluation of the Measurement and Evaluation in Education Curriculum. Stufflebeam (1999) stated that an effective program evaluation research that focuses on whether the program is successful is a comprehensive and expensive study that considers all questions and criteria. Today, this situation has revealed an understanding that considers the quality of the program evaluation process as well as the quality of the program (Astbury, 2016). This process, called meta-evaluation, is expressed as “evaluation of evaluation” (Scriven, 2009). Meta-evaluation draws attention in terms of determining who evaluates the program by considering which processes. While doing this, it is crucial to take various measures to prevent the standardization problem. Because it is necessary to decide which qualifications a program evaluation study should have through predetermined standards. Joint Committee on Standards for Educational Evaluation (JCSEE) at Western Michigan University conducts various researches to solve this problem. This organization, which has been working on educational standards since the 1980s, has developed the program evaluation standards, the 3rd edition of which has been already published (Fournier, 1994; JCSEE, 2018).
In program evaluation studies, it is appropriate to prefer approach, model, and methodological designs according to the program type and study context. On the contrary different studies examining program evaluation researches conducted in Turkey show that Stufflebeam's Context, Input, Process, and Product (CIPP) Evaluation Model is widely used in the evaluation of both teacher training programs and other programs (Kurt & Erdoğan, 2015; Özüdogru, 2018). This may be due to uncomplicated methodological choices in terms of executability in program evaluation studies carried out in Turkey. As a matter of fact, it is known that the CIPP Evaluation Model offers ease of application for different researchers (Stufflebeam & Coryn, 2014). In addition, it was stated that some program evaluation studies conducted in Turkey have various shortcomings (Akıncı & Köse, 2021). In this respect, it was considered crucial to describe all the evaluation procedures followed by the program evaluation study examined and to reveal the level of meeting the program evaluation standards.
This study is a meta-evaluation research. Stufflebeam (2000) defines meta-evaluation as the process of identifying, obtaining, and using descriptive and judgmental information about the usefulness, feasibility, relevance, and accuracy of that evaluation to publicly report on its strengths and weaknesses. For this purpose, the program evaluation standards prepared by JCSEE were taken into consideration during the meta-evaluation process. These standards contain thirty statements that define the five dimensions of the quality of program evaluation processes: utility, feasibility, propriety, accuracy, and accountability (JCSEE, 2018). The Program Evaluation Standards Checklist, developed using the relevant standards, was used to check whether the program evaluation research that was meta-evaluated met the required standards. During the preparation of the checklist developed by Akıncı and Köse (2020), opinions from different experts on translation, conceptual relevance, and item structure were obtained. Cohen Kappa coefficient of the agreement was calculated as 0.81 in terms of consistency between experts who examined the items. After these procedures, the final form of 30 items was given to the checklist by the researchers. Finally, an internal and an external evaluator checked whether the study had the right planning, valid and reliable data collection, necessary analyzes, and reporting qualities required for an effective program evaluation research. The descriptive analysis method was used in the analysis of the data collected through the checklist from internal and external evaluators.
Results / Expected Outcomes
The study, which was meta-evaluated according to the data obtained from internal and external evaluators, largely (88.33%) meets the Program Evaluation Standards determined by the JCSEE. According to the evaluators, all standard areas were met at a rate of 75% to 100%. However, experts believe that sufficient information is not provided in the research, especially about the concepts of cultural values, interests, contexts, and conflicts of interest. Akıncı and Köse (2020) stated that evaluation studies for teacher training programs in Turkey have similar shortcomings. The reason for this may be that cultural and contextual features are not considered sufficiently in program development studies carried out centrally in Turkey. Based on the results of the research, various suggestions that are thought to contribute to the quality of the meta-evaluated study and similar program evaluation studies and future program evaluation studies are presented. In this context, it is considered important that program evaluation studies should be carried out in a design that considers the appropriate approach, model and method choice, and contextual features.
Keywords: meta-evaluation, program evaluation, curriculum, teacher training.
Akıncı, M. & Köse, E. (2021). Research trends of program evaluation studies conducted between 2010-2019 in Turkey. Çukurova University Faculty of Education Journal, 50(1), 77-120.
Akıncı, M. & Köse, E. (2020). Türkiye’de Öğretmen Yetiştiren Programlara İlişkin Bir Meta-değerlendirme Çalışması. Uluslararası Pegem Eğitim Kongresi (pp.174-175). Diyarbakır, Türkiye.
Astbury, B. (2016). From evaluation theory to tests of evaluation theory?. In Stockmann, R. & Meyer, W. (Ed). The Future of Evaluation (pp. 309-325). Palgrave Macmillan, London.
Fournier, D. M. (1994). The program evaluation standards: How to assess evaluations of educational programs. Journal of Educational Measurement, 31(4), 363-367.
JCSEE. (2018). Checklist of the program evaluation standards statements. (2020, January 5). Retrieved from https://evaluationstandards.org/program.
Kurt, A. & Erdoğan, M. (2015). Content Analysis and Trends of Curriculum Evaluation Research: 2004-2013. Eğitim ve Bilim, 40(178), 199-224.
Özüdoğru, F. (2018). Analysis of curriculum evaluation studies conducted in foreign language education: 2005-2016. Journal of Language and Linguistic Studies, 14(2), 113-134.
Scriven, M. (2009). Meta-evaluation revisited. Journal of multidisciplinary evaluation, 6(11), 3-8.
Stufflebeam, D. L. (1999). Foundational models for 21st century program evaluation, The Evaluation Center Occasional Papers Series, Western Michigan University.
Stufflebeam, D. L. (2000). The methodology of metaevaluation. In Stufflebeam, D. L., Madaus, G. F., & Kellaghan, T. (Ed.) Evaluation Models: Viewpoints on Educational and Human Services Evaluation (pp. 457-471). Dordrecht: Kluwer Academic Publishers.
Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, and applications. (2nd Edition). San Francisco: Josey-Bass.