Evaluating Exam Providers
This tool can be used to assist regulatory administrators in evaluating the performance of exam providers. The following questions/topics may be used by regulatory administrators to evaluate the performance of a testing company.
General
- What evidence is present to confirm that the testing company adheres to commonly accepted testing industry standards in developing and administering the examination program?
- What background information exists about the testing company and the qualifications of the individuals who staff or will staff the program?
- What evidence is there that quality control procedures are incorporated into every aspect of the testing program? Is your agency assured that there are defined plans for business continuity so that your program can continue to operate without interruption in the case of an unexpected event?
- Do potential test takers know what qualifications, if any, the test taker must meet to be eligible to take the examination?
- Are there defined and documented processes for test takers to apply for special accommodations to test and for those accommodations to be reviewed and, if approved, provided to the test taker?
- What processes are defined so that all test taker and examination data are handled in a confidential and secure manner?
- How are test takers informed of their rights and responsibilities?
- Are policies in place to protect the financial aspects of the testing program along with those of the regulatory agency? Are financial audits undertaken to assure compliance with accounting guidelines?
Content
- Is there evidence that the testing program has been developed based on the foundation of a job analysis? If yes, is the job analysis updated on a regular basis?
- Do all test development activities undertaken relate back to the purpose and intent of the examination? Can all development activities be linked back to the job analysis?
- Are steps in place and carried out to keep the content refreshed on a timeline appropriate to changes in the profession or job for which the examination is focused?
- Does the sponsoring agency have access to the examination for review and approval purposes?
- Do test takers have a way to ask questions about examination content? What processes are defined to assess complaints or questions about content from test takers?
- Are examination items/content evaluated after administrations and are the reports of those evaluations used for future development activities?
- Are psychometric reports provided about examination items, examination forms and examination administrations?
Examination Administration
- Are administration policies documented for test takers?
- Are there written procedures that govern each examination administration to assure a secure administration?
- Are there policies for identifying and dealing with incidents of suspected or identified misconduct?
- How do test takers access information about the examination process including examination content, administration procedures, and how results will be reported?
- Are there documented procedures for identifying and resolving irregularities that occur in the administration process? If yes, is the information provided in a timely manner?
- Does the vendor pose reasonable or unreasonable barriers to changes in your policies and procedures?
Examination Scores
- Is there clear evidence that the determination of a pass or fail decision is made based on conducting a standard setting process?
- Are test takers informed about how examinations will be scored and what examination results mean?
- Is there an appeal process for candidates to challenge the examination results?
- Are users of test information informed about ways in which examination data can be appropriately used and any limitations about ways that examination scores cannot be used?
- Is there any evidence that the examination scores are being used for unintended purposes?
- Are technical reports provided to the regulatory agency to provide an official record of the administrations and results?