Frequently Asked Questions About Licensing Exams

Frequently Asked Questions About Licensing Exams

Linking occupational analysis to test items

CLEAR Exam Review (Summer 1996)
Norman R. Hertz

Question: We have completed an occupational analysis and used the results to develop a test plan for developing the examination. Doesn't our examination meet validity requirements?

Answer: Not necessarily. You have made a good start but if you want to establish the validity of your examination program, you must do more than conduct an occupational analysis and develop a test plan. A series of steps are required after the test plan has been developed. It is important to understand that an examination does not possess validity. Validity of a licensing examination is inferred if the examination tests the job-related competencies shown to be relevant from the results of an occupational analysis. Evidence for validity must be collected in the course of examination development, administration, scoring, and in establishing the passing score. If proper procedures are not applied consistently in any of these areas, then the validity of the examination may be questioned.

In the development of the examination, the first step is to ensure that the test questions are linked to the test plan and that the test plan, in turn, is linked to the results of the occupational analysis. Before the test questions are written, great care must be given in selecting the item writers to ensure that they represent the practice, at least in terms of ethnicity, gender, length of licensure, and specialty areas of practice. Otherwise, there may be a threat to the validity of the examination.
Further evidence for validity requires that the examination produce reliable results. Examinations that contain few questions, extremely difficult or easy questions, or questions that are flawed in some manner tend to produce unreliable results. An examination cannot produce valid results if it is not reliable.
Licensing examination questions must be written by item writers specially trained to write the questions that will discriminate among candidates who are qualified (minimally competent) versus those who are not.
Examinations that are compromised during administration pose one of the greatest threats to the validity of the examination. All the previous work is for naught if unauthorized persons gain access to the examination booklets prior to the examination and disseminate that information to candidates.
Tests are usually scored by scanning an answer sheet or by a computer program if the examination is computer administered. In both of these cases, the validity is threatened if the scoring key is incorrect. If the examination is scored by scanning answer sheets, then it is essential that quality control procedures be established and used to ensure that each answer sheet was accurately scanned.
Licensing examinations must have a passing score established at a level that represents minimal competence. An important validity check would be to obtain evidence that those who pass the examination are qualified to practice, while those who fail are not qualified. Hard data is extremely difficult to obtain and the best procedure is to ensure that the individuals establishing the passing score apply the criterion of minimal competence. Applying a fixed standard such as 75% is not an acceptable methodology for licensing examinations.

In summary, it is necessary to be vigilant at every step in an examination program in order to ensure the validity of the licensing decision. Validity is a function of doing many things correctly.