Credentialing organizations are expected to provide evidence of “reliability and validity” for any tests or assessment used in the credentialing process. Validity is considered of paramount importance—but what is it?
The concept of validity has been a topic of hot debate among psychometricians in terms of how it should be defined, conceptualized, communicated, and used. These issues bring a sparkle to the eyes of many psychometricians, but the eyes of others tend to glaze over as the discussion ensues. Fortunately, one does not need a psychometrician’s level of expertise in order to embrace the basic concept of validity. Validity has to do with whether a test serves its intended purpose effectively.
If you want to test basic math computation skills, obviously you would not use a reading comprehension test. The reading test would not be a valid measure of computation skills. But would you use a math test that included story problems or real-life situations? It depends. Do you just want to know whether your examinees can add two numbers? Or do you want to know whether they can determine, in a real-life situation, what mathematical operation is needed, select the two numbers, and add them correctly? A test that included story problems would probably be less valid as a test of basic computation than as a test of applied computation skills. Conversely, a test of straightforward computation problems would be less valid for measuring applied skills. Validity depends on the purpose of testing as well as the characteristics of the test itself.
The first step in ensuring a valid credentialing exam, then, is to clearly define the purpose of the exam. In both licensure and certification, the purpose of an exam is typically defined as protecting the public by ensuring that the credential is awarded only to those who have demonstrated that they have attained a certain level of knowledge and/or skill. The particular sets of knowledge, skills, and/or abilities to be tested are specific to the field or profession and are generally referred to as KSAs.
The next step is to design an exam that will serve the intended purpose by measuring the right set of KSAs. In licensure and certification, this is typically done by conducting a job analysis or practice analysis. Experts in the field define the job across a variety of settings in terms of the tasks performed and the KSAs needed to perform those tasks. Ideally, input is sought from practitioners at varying levels of experience. The outcome of the analysis is a test blueprint: a list of KSAs to be tested and the approximate proportion of the test that should address each KSA. A test constructed according to such a blueprint will measure the KSAs that are important for effective functioning in the field.
Designing a test on the basis of a solid job analysis or practice analysis helps to ensure that we are testing the right set of KSAs. Validity is further supported by constructing an exam according to sound measurement principles, so that we test the right set of KSAs effectively. This includes implementing good item writing practices, evaluating the performance of items statistically, ensuring that the test is long enough to provide reliable measurement, etc.
Once a credentialing exam has been designed and constructed to test the right set of KSAs effectively, it still may not serve its purpose if the passing standard is not set at an appropriate level of knowledge and/or skill. In credentialing, the score required to pass plays a very important role in validity. If the passing score is set too low, some people will be awarded a credential even though they have not attained the level of knowledge or skill that is expected of the licensed or certified practitioner. If the passing score is set too high, some applicants who do possess the required knowledge or skills will be shut out from the credential and its benefits. In either case, the exam fails to serve its purpose, and the meaning of the credential is changed.
There are many ways to establish additional evidence of validity. For example, test scores might be correlated with subsequent job performance evaluations. The steps listed above, however, are the basic steps most commonly used for establishing the validity of credentialing exams. If the purpose of the test has been clearly defined, if the test was designed and constructed in such a way as to measure the appropriate KSAs and measure them effectively, and if the passing score has been set at an appropriate level, it is likely that the test will serve its purpose effectively.
ERAC's Question and Answer Series is prepared by the CLEAR
Examination Resources and Advisory Committee (ERAC).
©2008 The Council on
Licensure, Enforcement and Regulation (CLEAR)