Frequently Asked Questions About Licensing Exams
Developing and pretesting items
CLEAR Exam Review (Summer
Eric Werner, M.A.
Question: My board invests a great deal of time and effort in developing new and valid questions for its written examination program. Every new item is subjected to strict scrutiny by subject matter experts before it is used. At the last CLEAR Conference, a person speaking on fundamentals of testing said that what we do is not enough because we don't "pretest" our questions. Why should we pretest in view of our already thorough review procedures?
Answer: Your question assumes that thorough review is possible without pretesting. I don't believe that it is. Without collecting and using experimental statistical information about how newly developed test questions actually function, subject matter expert review does not constitute sufficient analysis.
Expert judgment is most helpful when it is directed at a complete picture. Pretesting can help by providing statistical information about test item difficulty (experts often underestimate this), the extent to which particular items discriminate between higher- and lower-scoring candidates, the relationships among various parts of a proposed test, and reliability.
Pretesting is also a good way to identify item construction shortcomings and provides a basis for decisions to revise or delete certain items. Seldom do items function exactly as their developers expect they will and they should not be counted toward official examination results unless they survive some manner of pre-screening. This is especially important in relation to the goal of avoiding test bias, and it applies at least as much to oral, essay, and performance tests as to those in the multiple-choice format.
Administering a complete test form experimentally is often an infeasible and unnecessary approach to pretesting. At a minimum, however, a preliminary scoring and an item analysis review should be conducted to identify items of poor quality. After these items have been eliminated, a final scoring may be conducted. Although this is not pretesting in the literal sense, I believe that it can serve many of the same functions--assuming that not too many items have to be eliminated!
© 2002 Council on Licensure, Enforcement and Regulation