Inter-rater reliability calibration program: critical components for competency-based education
Measurement accuracy and reliability are important components of competency-based education (CBE). The ability to reliably assess student performance consistently over a period of time is crucial in these programs. For CBE programs which require multiple faculty or rater evaluations due to large student enrollments, it is expected that the faculty or raters score the student responses similarly. The reliability of ratings is critical for CBE programs because the integrity of the program rests on students demonstrating the specific competencies within the curriculum. Due to Western Governors University (WGU) large student enrollments, it employs several hundred evaluators to rate students performance assessments. WGU developed a program to assist their Evaluation Department in training their evaluators to score students assessments similarly. By using this program, WGU is able to maintain high inter-rater reliability rates within their evaluators and they are able to calibrate the ratings of their evaluators to ensure equitable and impartial ratings of assessments. This calibration has been very successful for new evaluator training and the retraining of current evaluators. WGU will continue to use this program in order to review and rate their assessments in an organized and systematic manner.
