Frequently Asked Questions About Licensing Exams

Computer-based testing

CLEAR Exam Review (Winter 1994)
Norman Hertz

Question: We are exploring the option of replacing our current paper-and-pencil test with a computer-administered version. What are the issues that we should address in making our determination?

Answer: Obviously, one should not automatically assume that the mode of examination administration, paper-and-pencil versus computer, is neutral in its effect on candidate performance. In going to computer-administered examinations, there should be some expected benefit to the candidate and the licensing agency. For the candidate, the examination might be offered more frequently, and at more sites; the pass/fail results might be provided at the completion of the examination; or less time might be required to complete the examination. The licensing agency may benefit by providing better service to candidates, having a more secure examination program, or improving accuracy of measurement.

A computer-based examination must be pilot-tested for obvious logistic reasons. However, the results from the pilot test should be examined to determine how its psychometric properties may differ from paper-and-pencil format. That is, one should be concerned that the medium in which the test is offered may have an effect on examination outcome. Particularly, the results should be examined to determine if the computer-based examination affected subpopulations of the candidates differentially when compared with paper-and-pencil administrations. The American Psychological Association's (1986) Guidelines for Computer-Based Tests and Interpretations states: "When interpreting scores from the computerized versions of conventional tests, the equivalence of scores from computerized versions should be established and documented before using norms or cutting scores from conventional tests." (p.18).

One would expect that well-constructed tests would not have an effect, regardless of the medium in which they were administered. Research has supported such a hypothesis for the most part, but there have been exceptions. If computer-based tests permitted candidates to review and change previous responses, results were comparable. For tests that were speeded and for tests that contained multi-screen items, the medium in which the test was offered had an effect. Speeded tests may also have an effect on candidate performance. (See Mead & Drasgow, 1993.)

Research tells us that carefully constructed power tests may be administered in a computer-based format without significant medium effects. Evidence is fairly supportive for adaptive examinations as well. Licensing boards should proceed with caution in planning to convert from paper-and-pencil to computer-based examinations. Given the potential benefits of computer-based examinations, however, it is worth-while for licensing boards to explore the opportunities.

Back to index

2002 Council on Licensure, Enforcement and Regulation