Frequently Asked Questions About Licensing Exams

Frequently Asked Questions About Licensing Exams

Criteria for a passing score

CLEAR Exam Review (Summer 1995)
Norman R. Hertz

Question: A nationwide association has responsibility for our examination program, including establishing the passing score. What options do we have other than accepting passing scores that have been established by the national association?

Answer: State licensing boards should retain the responsibility for establishing a passing score that reflects the standards required for minimal competence in their state. For many professions, the scope of practice does not vary significantly from state to state. However, performance expectations, training requirements, or extent of experience may vary and lead to higher or lower expectations in some states than in others. In such cases, the state licensing board may wish to establish its own passing score. In developing the contract, the licensing board should ensure that it retains responsibility for setting the passing score.

State licensing boards should obtain an explanation from the national association when a very high percentage of candidates either pass or fail the examination and the pass rate percentage differs substantially from the national passing rate. It would be best to work within the national association to influence the passing score process by involving practitioners from your state in developing the examinations and in establishing the passing score. However, if you licensing board believes that the passing score is not set at the appropriate level to provide for public protection, then it may be necessary for the board to establish its own passing score.

If a trade association is responsible for developing and marketing the examination, it should not establish the passing score. In such cases, a panel of practitioners -- independent of the association -- should establish the passing score to help allay accusations that entrance into the profession is restricted.

CLEAR Exam Review (Winter 1996)
Norman R. Hertz

Question: What are the key features I should look for in evaluating the quality of a criterion-referenced (e.g., Angoff) passing score study?

Answer: The credibility of the passing score can be questioned if all of the procedures are not properly carried out. Under the best conditions, the results from passing score workshops are likely to be challenged, so it is important to ensure that all steps are completed. The major steps are described below.

Selection of Workshop Participants
A diverse group of participants plays an important part in establishing a passing score that accurately reflects competence required for practice. The participants should have representation by specialty areas of practice, years of licensed practice, gender, and ethnicity. One of the most important population characteristics is years of licensed practice. Since the passing score is established at the level that represents minimum acceptable competence, it is important that at lease one-half of the participants have recently passed the licensing examination are likely to be more cognizant of the skills required for entry level than participants who have been licensed for many years. In selecting senior practitioners to serve as participants, be careful not to select practitioners who were granted licensure without taking the licensing examination.

Development of the Criterion
The term criterion-referenced passing score implies that the passing score is based upon a standard--a criterion. For licensing examinations, the criterion represents the level of candidate competence that demonstrates sufficient knowledge or skill to be able to practice safely. The question that is usually asked of the workshop participants is, "What percent of the minimally competent candidates would answer this question correctly?" The question cannot be answered reliably unless the parameters for the minimally competent candidate have been established.

The criterion of minimal competence should be established by having the workshop participants develop behaviorally based examples of performance expected from a minimally competent candidate. Do not assume that the participants have a common understanding of the performance expected of a minimally competent candidate.

Calibration of Participants
The participants must exercise their individual judgment during the passing score workshop. However, the participants should evaluate test questions consistently in terms of the percent of minimally competent candidates they believe would answer the question correctly. One should not expect the participants to evaluate the items identically, but the participants should apply the same standards in making their judgments.

Discussion of the Ratings
To maintain the integrity of the passing score workshop, independent ratings are a must. The importance of discussing the ratings becomes evident during the discussion when a participant brings up an issue that was unnoticed by the other participants. It is not uncommon that the new information greatly influences the ratings assigned by the other participants. Discussion of the ratings are also important when statistical information about the performance of the item is provided by the workshop leader. Participants may discuss and change their ratings when the statistical data about the difficulty of the item is different from their initial perception of the level of difficulty of the item.

Analysis of Passing Score Data
Data collected from the passing score workshops should by analyzed to estimate the reliability of the recommended passing score and to establish confidence intervals around the passing score. The mean (average) of the participants' ratings is the most representative score and should normally be used as the passing score. However, any score within the confidence interval may be used if sufficient rationale can be provided. Rationale for selecting a passing score other than the mean may include an extremely high or low mean, or consistently extreme ratings by some of the participants. Items vary in difficulty, so you should expect the participants' ratings would also reflect the variability.