Akademik Veri Yönetim Sistemi
Eğitim Programları ve Öğretim Kongresi (EPOK), İzmir, Türkiye, 4 - 06 Kasım 2021, (Yayınlanmadı)
Introduction
This study aims to conduct a meta-evaluation of the program evaluation
research titled Evaluation of the Measurement and Evaluation in Education Curriculum.
Stufflebeam (1999) stated that an effective program evaluation research that
focuses on whether the program is successful is a comprehensive and expensive
study that considers all questions and criteria. Today, this situation has
revealed an understanding that considers the quality of the program evaluation
process as well as the quality of the program (Astbury, 2016). This process,
called meta-evaluation, is expressed as “evaluation of evaluation” (Scriven,
2009). Meta-evaluation draws attention in terms of determining who evaluates
the program by considering which processes. While doing this, it is crucial to
take various measures to prevent the standardization problem. Because it is
necessary to decide which qualifications a program evaluation study should have
through predetermined standards. Joint Committee on Standards for Educational
Evaluation (JCSEE) at Western Michigan University conducts various researches
to solve this problem. This organization, which has been working on educational
standards since the 1980s, has developed the program evaluation standards, the
3rd edition of which has been already published (Fournier, 1994; JCSEE, 2018).
In program evaluation studies, it is appropriate to prefer approach,
model, and methodological designs according to the program type and study
context. On the contrary different studies examining program evaluation
researches conducted in Turkey show that Stufflebeam's Context, Input, Process,
and Product (CIPP) Evaluation Model is widely used in the evaluation of both
teacher training programs and other programs (Kurt & Erdoğan, 2015;
Özüdogru, 2018). This may be due to uncomplicated methodological choices in
terms of executability in program evaluation studies carried out in Turkey. As
a matter of fact, it is known that the CIPP Evaluation Model offers ease of
application for different researchers (Stufflebeam & Coryn, 2014). In addition, it was stated that some program evaluation
studies conducted in Turkey have various shortcomings (Akıncı & Köse, 2021).
In this respect, it was considered crucial to describe all the evaluation
procedures followed by the program evaluation study examined and to reveal the
level of meeting the program evaluation standards.
Methodology
This study is a meta-evaluation
research. Stufflebeam (2000) defines meta-evaluation as the process of
identifying, obtaining, and using descriptive and judgmental information about
the usefulness, feasibility, relevance, and accuracy of that evaluation to
publicly report on its strengths and weaknesses. For this purpose, the program
evaluation standards prepared by JCSEE were taken into consideration during the
meta-evaluation process. These standards contain thirty statements that define
the five dimensions of the quality of program evaluation processes: utility,
feasibility, propriety, accuracy, and accountability (JCSEE, 2018). The Program
Evaluation Standards Checklist, developed using the relevant standards, was
used to check whether the program evaluation research that was meta-evaluated
met the required standards. During the preparation of the checklist developed
by Akıncı and Köse (2020), opinions from different experts on translation,
conceptual relevance, and item structure were obtained. Cohen Kappa coefficient
of the agreement was calculated as 0.81 in terms of consistency between experts
who examined the items. After these procedures, the final form of 30 items was
given to the checklist by the researchers. Finally, an internal and an external
evaluator checked whether the study had the right planning, valid and reliable
data collection, necessary analyzes, and reporting qualities required for an
effective program evaluation research. The descriptive analysis method was used
in the analysis of the data collected through the checklist from internal and
external evaluators.
Results / Expected
Outcomes
The study, which was meta-evaluated according to the data obtained from
internal and external evaluators, largely (88.33%) meets the Program Evaluation
Standards determined by the JCSEE. According to the evaluators, all standard
areas were met at a rate of 75% to 100%. However, experts believe that sufficient
information is not provided in the research, especially about the concepts of
cultural values, interests, contexts, and conflicts of interest. Akıncı and
Köse (2020) stated that evaluation studies for teacher training programs in
Turkey have similar shortcomings. The reason for this may be that cultural and
contextual features are not considered sufficiently in program development
studies carried out centrally in Turkey. Based on the results of the research,
various suggestions that are thought to contribute to the quality of the
meta-evaluated study and similar program evaluation studies and future program
evaluation studies are presented. In this context, it is considered important
that program evaluation studies should be carried out in a design that
considers the appropriate approach, model and method choice, and contextual
features.
Keywords:
meta-evaluation, program evaluation, curriculum, teacher training.
References
Akıncı, M. & Köse, E. (2021). Research
trends of program evaluation studies conducted between 2010-2019 in Turkey. Çukurova
University Faculty of Education Journal, 50(1), 77-120.
Akıncı, M. & Köse,
E. (2020). Türkiye’de Öğretmen Yetiştiren Programlara İlişkin Bir
Meta-değerlendirme Çalışması. Uluslararası Pegem Eğitim Kongresi
(pp.174-175). Diyarbakır, Türkiye.
Astbury,
B. (2016). From evaluation theory to tests of evaluation theory?. In Stockmann,
R. & Meyer, W. (Ed). The Future of Evaluation (pp. 309-325).
Palgrave Macmillan, London.
Fournier, D. M. (1994).
The program evaluation standards: How to assess evaluations of educational
programs. Journal of Educational Measurement, 31(4), 363-367.
JCSEE. (2018).
Checklist of the program evaluation standards statements. (2020, January 5). Retrieved
from https://evaluationstandards.org/program.
Kurt, A. & Erdoğan, M. (2015). Content Analysis
and Trends of Curriculum Evaluation Research: 2004-2013. Eğitim ve Bilim,
40(178), 199-224.
Özüdoğru, F. (2018). Analysis of
curriculum evaluation studies conducted in foreign language education:
2005-2016. Journal of Language and Linguistic Studies, 14(2), 113-134.
Scriven, M. (2009).
Meta-evaluation revisited. Journal of multidisciplinary evaluation, 6(11),
3-8.
Stufflebeam, D. L. (1999). Foundational models for 21st century program evaluation, The
Evaluation Center Occasional Papers Series, Western Michigan University.
Stufflebeam, D. L.
(2000). The methodology of metaevaluation. In Stufflebeam, D. L., Madaus, G.
F., & Kellaghan, T. (Ed.) Evaluation
Models: Viewpoints on Educational and Human Services Evaluation (pp.
457-471). Dordrecht: Kluwer Academic Publishers.
Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, and applications.
(2nd Edition). San Francisco: Josey-Bass.