Frequently Asked Questions About Licensing Exams

Item analysis theory

CLEAR Exam Review (Summer 1995)
Norman Hertz

Question: In our item writing workshops, we develop quite a few items that do not survive pretesting based on item analysis results. What can we do to ensure that more of the items will survive?

Answer: There are no shortcuts for producing quality items. Developing items is time consuming and, of course, costly-the cost per item is driven up because many of the items do not survive review and pretesting. However, there are some procedures, if followed, that will improve the quality of the items and increase the number of usable items. Flawed items should be removed early in the development process to reduce the costs involved with processing them. The longer the items stay in the development process, the more likely they are to survive even if they are flawed.

The first step is to develop the items according to a test blueprint. The test blueprint provides a detailed description of the subject matter areas that should be included in the examination. Close adherence to the test blueprint will increase the number of items that survive because the items are likely to be viewed as relevant.

Another important step is to ensure that the item writers are demographically representative of the practitioners. If the item writers represent the diversity of the practitioner population, biased items are unlikely to survive the initial item-writing workshop.

Also, the item writers should understand that a critique of their items by participants in the workshop is essential to ensure that items are universally true, that they do not reflect individual preference or a narrow application. The process of initial review is enhanced if the items are written in draft form with the understanding that, regardless of the strengths of item writers, items can be improved. Finally, a group of practitioners who did not participate in the original item writing should review the items before they are pretested to make final edits or to remove any items they judge to be flawed.

In summary, the cost of developing items can very easily amount to several hundred dollars or more per item, so it is important that the items placed in the item bank are sound. The earlier in the process questionable items are repaired, edited, substantially revised, or removed, the more likely fewer items will be rejected during pretesting or after they are used in an actual examination.


Back to index

2002 Council on Licensure, Enforcement and Regulation