Frequently Asked Questions About Licensing Exams

Limiting test references

CLEAR Exam Review (Winter 1997)
Norman R. Hertz

Question: I understand the need to base test questions on reference materials. However, in many professions, the potential list of references is extremely large. Should there be a limit to the number of references on which a test is based? What kind of references are suitable?

Answer: For several reasons, I think it is wise to identify a limited list of suitable references before test question writing begins. It is reasonable to inform candidates that the examination will be taken from a list of references. Potential problems are reduced if there is a finite list of references. For example, conflicts in references, or old or obsolete references, are less likely to occur if there is a limitation on the date of publication. Also, if journals are used as references, it would be better to use only refereed journals and only those journals that truly represent the field. In some fields, there is a major theme or a preferred methodology as well as a peripheral one. Unless there is a restriction in the references, the better candidates may be disadvantaged because of their broad awareness of different approaches, and they may be forced to choose from their perception of several correct answers.

One manifestation of using too many or outdated references may be a variability in item statistics for different examination administrations. Another indication may be a variability in the percent of candidates who pass a function of the examination form administered. To get a sense of the number and types of suitable references for an examination, a methodology must be proposed. For example, one could contact training institutions and professional associations within the state. A list could be compiled from suggestions by these sources, then reviewed by subject matter experts who could also suggest the final reference list. A note of caution is in order--just because books or journals are used in training institutions does not mean they are appropriate references for licensing examinations.

Back to index

2002 Council on Licensure, Enforcement and Regulation