Developing Criterion-Referenced Tests
By Elham Arabi, PhD
Quizzes having little to do with performance of skills with arbitrary cut-off scores do not determine a person’s success in a training. This means they are not criterion-referenced, which means tests that measure a person’s performance against a set of knowledge and skills.
So, while it is important to have robust assessment methods that can measure a learner’s ability to perform the acquired skills, it is even more vital (and usually more challenging) to assess learners in high-stakes settings, such as safety, healthcare, or aviation in a more robust manner.
These are some key steps for developing criterion-referenced tests that give meaningful data:
Analyze how tasks are performed on the job.
Determine the metrics (essential knowledge, decision-making, task performance)
Decide how you’re going to measure (tests, checklists or raters to demonstrate the skills)
Develop performance objectives aligned with the task(s) that learners need to perform on the job (use Sugrue’s guide for writing performance objectives)
Repeatedly refer to the skills and objectives when developing tests or checklists – it should be an iterative process
Do a validity test to make sure the test items measure what they’re supposed to measure by mapping each item with the performance objectives
Set the appropriate cut-off score (the more common and feasible one is Bookmark method) – There are other methods that can be used depending on an organization’s resources
Follow the guideline on writing good and performance-focused MCQs (Check out this resource but there are many available ones on the web)