Frequently Asked Questions About Licensing Exams

Frequently Asked Questions About Licensing Exams

Content validation of examinations

CLEAR Exam Review (Summer 1995)
Norman R. Hertz

Question: What does validity mean in terms of licensing and certification examinations? If an examination is valid, does it mean that candidates who score well on the examination will perform more effectively on the job than a candidate who scores less well?

Answer: The validity referred to in licensing examinations is known as content-related validity. An examination is considered content valid if it is based upon the results of an occupational analysis, sometimes called a job analysis or practice analysis. Content-related validity if based on the premise that a candidate who passes a licensing examination is knowledgeable in the required content of the job. It is an inappropriate interpretation to use the results to predict how effective a candidate will be in the occupation.

In employment contexts, predictive validity can be established demonstrating that a higher score on employment examinations is associated with higher performance on the job. No such relationship can be established in licensing settings. All who pass the licensing examination are considered competent to practice. Examinations of this type are used to make mastery and non-mastery decisions. Any other use of the scores within these categories is inappropriate.

CLEAR Exam Review (Summer 1996)
Norman R. Hertz

Question: In what ways do licensing examinations differ from academic (achievement) and employment examinations?

Answer: First, examinations are designed with regard to their purpose. Licensing examinations are designed to assess candidates' competence after they have completed their education or training and obtained experience. Furthermore, licensing examinations should be designed to assess higher level skills; that is, examinations should not only assess candidates' knowledge, but also whether they are able to apply that knowledge to solve problems or handle situations that they may face in actual practice.

In contrast, academic examinations are designed to measure students' knowledge of material presented in the classroom. The focus of the examination is usually relatively narrow and covers only the material learned in the class--much narrower than the scope of licensing examinations.

Employment examinations are broader in scope than academic examinations but narrower than licensing examinations. Employment examinations are designed to meet job-related standards, as are licensing examinations. However, employment examinations are designed to predict whether a test taker who scores high on an examination will perform well on the job. It is important to make the distinction here that licensing examinations should not be used as a substitute for employment examinations.

The focus of licensing examinations is not to predict success as an employment examination might, but to assure the public that the person who is licensed is qualified to practice without harming the public. By design, licensing examinations do not measure skills critical to business success such as knowledge of office procedures or management practices.

For the qualified candidate who has obtained the education and quality experience, the licensing examination should not be an insurmountable hurdle. The licensing examination is designed to assess candidates' ability to apply the competencies that they gained from their education and experience. Since the examination is applied rather than theoretically based, the licensing examination is much different from a final examination in a college course.

The consequences of an improperly developed licensing examination are much greater than those of an academic or employment examination. If the licensing examination is designed so that unqualified candidates pass the examination the public may be harmed. The costs associated with an improperly developed academic examination may be that the student received a lower grade than was deserved, and for an improperly developed employment examination, there may be a delay in employment. However, improperly developed licensing examinations may result in a candidate's inability to work in his or her desired occupation or profession.

CLEAR Exam Review (Summer 1996)
Norman R. Hertz

Question: We have completed an occupational analysis and used the results to develop a test plan for developing the examination. Doesn't our examination meet validity requirements?

Answer: Not necessarily. You have made a good start but if you want to establish the validity of your examination program, you must do more than conduct an occupational analysis and develop a test plan. A series of steps are required after the test plan has been developed. It is important to understand that an examination does not possess validity. Validity of a licensing examination is inferred if the examination tests the job-related competencies shown to be relevant from the results of an occupational analysis. Evidence for validity must be collected in the course of examination development, administration, scoring, and in establishing the passing score. If proper procedures are not applied consistently in any of these areas, then the validity of the examination may be questioned.

In the development of the examination, the first step is to ensure that the test questions are linked to the test plan and that the test plan, in turn, is linked to the results of the occupational analysis. Before the test questions are written, great care must be given in selecting the item writers to ensure that they represent the practice, at least in terms of ethnicity, gender, length of licensure, and specialty areas of practice. Otherwise, there may be a threat to the validity of the examination.
Further evidence for validity requires that the examination produce reliable results. Examinations that contain few questions, extremely difficult or easy questions, or questions that are flawed in some manner tend to produce unreliable results. An examination cannot produce valid results if it is not reliable.
Licensing examination questions must be written by item writers specially trained to write the questions that will discriminate among candidates who are qualified (minimally competent) versus those who are not.
Examinations that are comprised during administration pose one of the greatest threats to the validity of the examination. All the previous work is for naught if unauthorized persons gain access to the examination booklets prior to the examination and disseminate that information to candidates.
Tests are usually scored by scanning an answer sheet or by a computer program if the examination is computer administered. In both of these cases, the validity is threatened if the scoring key is incorrect. If the examination is scored by scanning answer sheets, then it is essential that quality control procedures be established and used to ensure that each answer sheet was accurately scanned.
Licensing examinations must have a passing score established at a level that represents minimal competence. An important validity check would be to obtain evidence that those who pass the examination are qualified to practice, while those who fail are not qualified. Hard data is extremely difficult to obtain and the best procedure is to ensure that the individuals establishing the passing score apply the criterion of minimal competence. Applying a fixed standard such as 75% is not an acceptable methodology for licensing examinations.

In summary, it is necessary to be vigilant at every step in an examination program in order to ensure the validity of the licensing decision. Validity is a function of doing many things correctly.