Writing pretest items/ writer workshops

CLEAR Exam Review (Summer 1996)
Norman Hertz

Question: We have been asking item writers to prepare questions in advance of the item writing workshop. However, we have been told that while it seems like a reasonable process, it may be inadvisable. What are the potential problems?

Answer: I am aware that a number of test providers use such a procedure. However, there are a number of distinctions that should be drawn between large and small examination programs.

Security. First, the procedures may be more suited for a large-scale examination program since typically the items produced during the item writing workshop are stored in a large item bank and probably will not all be used on the same version of an examination. Therefore, if the security for all the items of one item writer were compromised, the overall security of the examination would be maintained, since only a few items from that item writer are likely to have been included in the examination, and these more than likely would have been modified. It should be noted, however, that if some candidates have unfair access to only two or three test questions, that could be the difference between passing or failing the examination. On the other hand, if an examination program does not have a large item bank or if a licensing program is new, it is conceivable that many of the compromised items would appear on the examination thereby causing its results to be challenged.

Most item writers are responsible and are interested in keeping the items secure. However, when the items are written in a setting that is not secure, it is possible that unauthorized persons could gain access to the items. Some agencies using outside item writers require that they sign security agreements promising to keep the items secure. It is important to weigh the benefits of having the items prepared in advance with the potential costs if the integrity of the examination were challenged.

Quality. If items are written by persons who have not been trained specifically to write items for licensing examinations, many of the items will not survive a critical review. The procedures for writing questions for licensing examinations are sufficiently different from the procedures for writing multiple-choice questions for academic-type examinations. For item writers whose experience is limited to those types of examinations, additional training is necessary. If the items have to be significantly reworked, which invariably is the case, the benefit of bringing items to the workshop is marginal at best.

Back to index

© 2002 Council on Licensure, Enforcement and Regulation