CLEAR Exam Review (CER) is a journal published twice a year that reviews issues affecting testing and credentialing. Subscriptions to CER are sent free of charge to all CLEAR members and are available for $30 per year to others. Advertisements and classifieds (e.g. position vacancies) may be reserved by contacting CLEAR at (859) 269-1701. Ads are limited in size to 1/4 or 1/2 of an 8 1/2 x 11 page, and cost $100 or $200, respectively, per issue.
Editing and composition of this journal have been underwritten by The Chauncey Group International, which specializes in the design, development and implementation of high-quality licensing and certification programs.
"Testing Across the Nation" by
Sandra Greenberg and Karen Cullen
Greenberg and Cullen focus on innovative uses of technology in credentialing programs. They discuss the use of electronic data in a practice analysis study, Web-based self-assessment examinations, online application processing, Web access to credential information, and online license renewals.
"Abstracts and Updates" by Charles Kunce
and Mary Sandifer
Kunce and Sandifer discuss recent articles, abstracts and presentations on differential item functioning (DIF), performance assessment, and computers and testing.
"Software Review" by Lee Schroeder
Schroeder's reviews IMSI's HiJaak Pro Version 5, a software product that allows the user to convert, capture and organize graphic files. This should mean that graphics can be stored and used more easily in test questions.
"Legal Beat" by Dale Atkinson
Atkinson discusses a recent case that focuses on the appropriateness of testing agencies and organizations disclosing to recipients of test scores that the examination was administered under nonstandard conditions. The opinion in the case analyzed Title III of the Americans with Disabilities Act (ADA) as it relates to private entities providing examinations to the public.
"Testing and
Measurement Issues" by Thomas Henzel and Kristina Golden
Henzel and Golden focus un the the structural complexity of test items for computer-based testing. Their column identifies a new classification code that describes a test item in terms of its structural complexity. They suggest this new code may be useful in the reproduction of multiple examination forms that are balanced for content and difficulty, as well as complexity of item construction.
"Defining the Scope of a
New Job and its Knowledge Base: The Application of a Job Analysis Study to a
Decision-Making Process" Michael Rosenfeld, Sarah Slater and Sharon
Goldsmith
The authors describe how job analysis techniques may be used to define the scope of a new job, assist in defining the curriculum necessary to perform the tasks and acquire the skills necessary for competent entry-level performance, and serve as a basis for standards to be used to credential approved training programs.
"Condensed
Job Analysis: Capturing a Moving Target in the Information Technology
Field" by James Adair
Adair discusses the development of an alternative job analysis method. It focuses particularly on the adaptation of existing job analysis techniques and processes to meet the needs of the Information Technology industry for assessing quickly altering job and product-related content.
"Testing Across the Nation" by Jim
Zukowski
Zukowski discusses an audit program recently approved in the state of California that will impact upon credentialing examination programs related to professional licensure. He also reviews recent U.S. Supreme Court decisions that will affect the application of ADA with regards to testing accommodation requests, provides an update on a job analysis for social workers, and describes a new certification program for information technology specialists.
"Abstracts and Updates" by Charles Kunce
and Mary Sandifer
Kunce and Sandifer discuss recent articles, abstracts and presentations on test adaptation, computer-based testing, and the impact of new technology on examinations. Other issues include gender differences, validation and performance testing and the accuracy of item ratings of subject matter experts.
"Software Review" by Lee Schroeder
Schroeder's reviews the Statsoft Electronic Statistics Textbook, a book of reference information that can be accessed for free on the Internet. He recommends this particularly for those that find themselves in need of information on statistics when out of the office.
"Legal Beat" by Dale Atkinson
Atkinson discusses a case of a dentist with a limited license from the state of Arizona. After the legislature repealed the section of legislation providing for limited licenses, the dentist filed a complaint for violation of his due process and equal protection rights.
"Testing and
Measurement Issues" by Gerard Dillon and William Walsh
Dillon and Walsh focus on using performance data to set standards, its practical impact and the perception of judges. They also summarize the results from a research study that they recently conducted.
"On the Documentation of
Credentialing Examination Procedures and Policies" by James Fidler
The author considers issues pertinent to the preparation of performance and policies manuals for credentialing examination programs. The issues discussed by Fidler include: the importance and uses of documentation; contents and format of a manual; appropriate author and audience; and document storage.
"A
Brief Discussion of Three Basic Computer-Based Testing Models" by
Jerry
Gorham and Jian Zhang
Gorham and Zhang provide an introduction to computer-based testing and discuss the different models available.
"Testing Across the Nation" by Jim
Zukowski
Zukowski discusses a new partnership between the Chauncey Group and the NorthWest Center for Emerging Technology (NWCET) concerned with the Information Technology industry. Also mentioned are the revised Standards for Educational and Psychological Testing, an update on the practice analysis in orthotics and prosthetics, computer-based podiatry testing, the development of a set of guidelines for computer-based testing, and new standards for food protection managers.
"Abstracts and Updates" by Charles Kunce
and Mary Sandifer
Kunce and Sandifer discuss recent articles, abstracts and presentations on educational standards, computer-based testing (item exposure, automated test assembly, and user interface), ADA accommodations, performance assessment, equating and scaling, and job analysis.
"Software Review" by Lee Schroeder
Schroeder's article focuses on Item Response Theory (IRT) and provides a review of the theory, before considering a software product (XCALIBRE) which can be used to calculate IRT parameters.
"Legal Beat" by Dale Atkinson
Atkinson discusses a recent Missouri case that focused on the reliance of regulatory boards on licensure examinations and whether limitations to this authority exist.
"Testing and
Measurement Issues" by Brian Klauser and Ronald Nungester
Klauser and Nungester focus on the factors that need to be considered when identifying cut scores in certification and licensure decisions.
"Adapting Credentialing
Exams for Use in Multiple Languages" by Ronald Hambleton, Stephen Sireci,
and Frederic Robin
The authors consider reasons for adapting examinations for use in other languages and steps that can be taken to adapt the examinations. Eight steps are recommended: checking exam content and format equivalence; deciding the desirability of a translation; choosing translators; translating and adapting the exam; reviewing the adapted version of the exam; conducting a small tryout of the adapted exam; carrying out a more ambitious study of the adapted exam; and documenting the process.
"Testing Across the
Nation" by Jim ZukowskiZukowski discusses a number of topics that are currently being considered by several state and national boards. His column discusses such issues as the transition to computer-based testing, examination ownership, and multi-state job analysis.
"Testing,
Testing...1, 2, 3" by Grady Barnhill and Lynn WebbBarnhill and Webb talk about credentialing agency responsibilities to the failing candidate. They discuss the pros and cons of providing diagnostic feedback to candidates.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer discuss the status of both the revised Standards for Educational and Psychological Testing and the "Rights and Responsibilities of Test Takers". Additionally they review articles dealing with the detection of cheating, consequential validity, item relevance, learning disabilities, standard setting, and essay and performance test scoring.
"Software Review"
by Lee SchroederSchroeder discusses concerns about the security of documents that are sent using e-mail or through the Internet. Lee describes software that uses public key cryptography to help solve this problem.
"Legal Beat" by
Dale AtkinsonAtkinson discusses how managed care can make issues of defining scope of practice and determining jurisdiction quite challenging to regulatory boards.
"Testing and Measurement
Issues" by Shelby Keiser
Keiser discusses factors licensing boards should consider when responding to individuals requesting test accommodations. She emphasizes the fact that licensing agencies are not educational institutions. Their mission is public protection.
"Security for CBT High
Stakes Licensure Exams" by Barbara HalseyHalsey discusses the way in which the National Council of State Boards of Nursing attempts to maintain the security of its computer administered licensing examinations. These include such aspects as data transmission and back-up, software security, the training and certification of administrators/proctors, identification/check-in procedures, confidentiality agreements, and the testing environment.
"Guidelines for
Selecting a Standard Setting Panel for Licensure Testing" by David M.
WilliamsonWilliamson discusses factors to consider when selecting participants in a standard setting panel. He emphasizes that these factors are important in order to ensure the integrity of the standard setting process.
"Testing Across the
Nation" by Jim ZukowskiZukowski discusses methods related to standard setting for performance assessments and describes a new method developed by the Education Commission for Foreign Medical Graduates.
"Testing,
Testing...1, 2, 3" by Grady Barnhill and Lynn WebbBarnhill and Webb talk about all the things you need to consider when facilitating meetings. They discuss how to encourage subject matter experts to attend meetings, the ideal group size, and the role of the facilitator.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer discuss articles dealing with political and regulatory issues, performance testing, and standard setting methods.
"Software Review"
by Lee SchroederSchroeder talks about a problem of concern to many of us - the security of our computers. Lee discusses software that is available to protect the security of your files in the event your laptop is lost or stolen.
"Legal Beat" by
Dale AtkinsonAtkinson discusses one of the major functions of legislatively created regulatory boards - the conduct of disciplinary proceedings against licensees accused of wrongdoing. He discusses the standards of proof necessary for a board to establish wrongdoing of an accused licensee.
"Testing and
Measurement Issues" by Richard LuechtLuecht discusses how moving from paper-and-pencil testing to computerized testing can require fundamental changes in the way tests are produced. He describes an automated test assembly approach that helps meet increased item development needs.
"Job Analysis for
High-stakes Credentialing Examinations" by Vicki L. Flaherty and James B.
HoganFlaherty and Hogan discuss the role job analysis plays in test development and validity in credentialing examinations. They discuss a number of practical issues concerning how to plan, conduct, analyze, and report on the results of job analysis studies.
"The National Board of
Podiatric Medical Examiners New Testing Methodology" by Charles Gibley, Jr.Gibley describes how the testing program conducted by the National Board of Podiatric Medical Examiners evolved over several years from a paper-and-pencil program to one that is based on computerized mastery testing (CMT). The article describes the steps that were taken to move from paper-and pencil tests to linear computer-delivered tests, and then on to CMTs.
"Testing Across the
Nation" by Jim ZukowskiZukowski discusses the "Guidelines for Documentation of a Learning Disability in Adolescents and Adults," developed by the Association on Higher Education and Disability (AHEAD). In addition to their use in higher education, these guidelines have important implications for candidates who are are applying for test accommodations for licensing and certification examinations. Zukowski also describes the procedures used by the National Institute for Hearing Instrument Studies to develop their international licensing examination for hearing instrument dispenser licensing boards.
"Testing,
Testing...1, 2, 3" by Grady Barnhill and Lynn WebbBarnhill and Webb talk about how the delivery modes for assessing professional competency--ranging from oral exams to virtual reality testing--have changed over time. Their timeline extends from 2357 B.C. to the present. The conversation makes clear that some delivery modes are better geared to the achievement of particular assessment objectives than others.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer discuss articles dealing with different language and cross-cultural testing, bias and differential item functioning, significance testing, consequential validity, innovative uses of computers in testing, and performance testings.
"Software Review"
by Lee SchroederSchroeder reviews a relatively inexpensive statistical package that he found to be very comprehensive and useful.
"Legal Beat" by
Dale AtkinsonAtkinson looks at two opposing judicial opinions on the same issue to remind board members that judicial opinions may differ on the same issue, not only from state to state, but also from district court to district court, or from circuit court to circuit court. These differing opinions make a board member's job difficult and challenging.
"Testing and
Measurement Issues" by Gerard DillonDillon describes the use of survey data in a testing program. He discusses how asking test takers focused questions about their testing experience can provide useful information for modifying and improving various aspects of a testing program.
"Advances in the Use
of Item Response Theory in Licensure and Certification Assessment" by Betty
A. Bergstrom and Richard C. GershonBergstrom and Gershon outline the differences between classical test theory and item response theory (IRT) and describe how applications of IRT are relevant to regulatory and credentialing organizations. They discuss the use of IRT for item banking, automated test construction, online test construction, computerized adaptive testing, computer-based simulations, performance assessment and surveys.
"'Mental Model
Comparison of Automated and Human Holistic Scoring of Architectural Design
Simulations" by David M. Williamson, Issac I. Bejar and Anne HoneThe authors compare automated scoring with human scoring. Using data gathered from the Architect Registration Exam (ARE), they compare the computer-based scoring of open-ended architectural problems with the results obtained from human scorers.
"Testing Across the
Nation" by Jim ZukowskiZukowski describes the new Multistate Pharmacy Jurisprudence Examination, a take-home recertification examination developed by the American Board of Internal medicine. He also examines the Internet as a way to provide information about licensing to candidates and the public.
"Testing,
Testing...1, 2, 3" by Grady Barnhill and Lynn WebbBarnhill and Webb provide useful information about assessing continuing competence in a lively, conversational manner.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer review a new book published by CLEAR, Demystifying Occupational and Professional Regulation: Answers to Questions You May Have Been Afraid to Ask, by Kara Schmitt and Ben Shimberg. Next, they cover a special credentialing issue of the journal Applied Measurement in Education, followed by a list of sessions at the 1997 meetings of the American Educational Research Association and the National Council on Measurement in Education that dealt with issues important to credentialing agencies.
"Software Review"
by Lee Schroeder and Pansy HoughtonSchroeder and Houghton discuss a product that allows for in-house design and printing of double-sided, scannable forms.
"Legal Beat" by
Dale AtkinsonAtkinson discusses two cases involving a board's authority to investigate the activities of licensees. In these two cases, the board's authority was upheld--but these processes may not be protected in all jurisdictions.
"Testing and
Measurement Issues" by Anthony LaDucaLaDuca describes the approach the National Board of Medical Examiners is taking to evaluate practicing physicians' continued competence, a topic of interest for many professions.
"New Testing
Methodologies for the Architect Registration Examination" by Jeffrey F.
KenneyKenney explains new testing methodologies being used by the National Council of Architectural Registration Boards. In particular, the article describes the development of procedures that enable a computer to score and administer small, focused design problems, known as "vignettes," which simulate an architect's design work.
"Is the Angoff Method
Really Fundamentally Flawed?" by Michael ZiekyZieky asks whether or not the Angoff method used to set passing scores is fundamentally flawed, as recently alleged. This article is particularly useful for those who use this standard-setting procedure.
"Testing
Across the Nation" by Barbara ShowersShowers makes a plea for the provision of diagnostic information to candidates who fail credentialing examinations. She discusses the psychometric considerations as well as some creative solutions that can be used to provide candidates some of the information they seek.
"The Answer Key"
by Norman HertzHertz responds to four common examination questions. The first deals with the ideal number and type of references upon which to base an examination. The second concerns possible problems associated with a state adopting an examination developed and administered by an association of state boards. The third responds to a concern raised by a small licensing program that wants to ensure its assessments comply with national standards. The fourth deals with the grounds on which appeals of examination results should be considered.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer review a new book on computer-based testing, six references dealing with general certification issues, three references concerned with the Americans with Disabilities Act, as well as six articles devoted to issues and viewpoints pertaining to performance assessment.
"Software Review"
by Lee Schroeder and Pansy HoughtonSchroeder and Houghton discuss software that can be used to manage bands of examination questions. Their focus is on software that can be used to ensure that duplicate or very similar items are not unknowingly included in the item bank.
"Legal Beat" by
Dale AtkinsonAtkinson discusses whether or not the exam scores of licensees should be made available to the general public.
"Testing and
Measurement Issues" by Janice ScheunemanScheuneman provides an objective and balanced discussion of the advantages and disadvantages of computer-based testing.
"Using Nominal Group
Technique (NGT) to Identify Factors Potentially Influencing a Consistently Low
Passing Rate" by Linda M. DeanDean describes the use of the Nominal Group Technique to identify factors potentially influencing a consistently low passing rate. It incorporates a process borrowed from the research on group dynamics to solve a testing problem encountered in licensure.
"Testing Across the
Nation" by Barbara ShowersShowers explores whether or not credential test providers are performing reliability analyses at the cut score to determine which, if any, of the analyses have become "state of the art" among providers. The data for this column were collected via a survey sent to credential test providers.
"The Answer Key"
by Norman HertzHertz answers four common examination questions. The first relates to the advisability of asking item writers to prepare questions in advance of the item writing workshop. The second deals with how licensing and certification tests differ from academic (achievement) and employment examinations. The third concerns whether the completion of an occupational analysis and the setting of test specifications are sufficient to meet validity requirements for an examination used in licensing and certification. The fourth deals with how to explain passing score variation from one administration to another to candidates when using a criterion-referenced methodology.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer describe a new reference book on licensure testing, five articles discussing the use of computers to grade performance tests, an article on rater accuracy in performance testing, three articles about guidelines for translating exams, and two articles about candidate reactions to new licensure tests.
"Software Review"
by Lee Schroeder and Pansy HoughtonSchroeder and Houghton discuss the strengths and weaknesses of introducing technology to reduce or even substitute for the need to have face-to-face meetings to review test questions. The column describes the use of a modem to conduct a conference over a single telephone line that shares voice and computer data simultaneously.
"Legal Beat" by
Dale AtkinsonAtkinson describes the legal concept of collateral estoppel and its implications for administrative proceedings against licensees in regulated professions. He also discusses a case in United States District Court, Northern District, involving the applicability of Title VII of the Civil Rights Act to voluntary certification agencies, a first of its kind.
"The Expectations of
Standard Setting Judges" by Gerard F. DillonDillon discusses how the results of standard setting can sometimes be viewed as unacceptable by relevant stakeholders. He describes a technique he believes can be used to reduce this problem that can also be helpful in training judges participating in standard setting exercises.
"Item Harvesting: An
Efficient Method for Generating Test Questions" by John Norcini, Paul
Poniatowski, Susan Day, and Elizabeth CallahanThe authors describe a procedure, called "item harvesting," for producing test questions. The procedure solicits the active participation of large numbers of practitioners, is carried out by mail and does not require a large-scale, face-to-face test committee meeting to review all of the questions.
"Test Validity
Evidence: What About Face Validity?" by Steven M. DowningDowning discusses the concept of face validity and its misuse, particularly in performance testing. This article emphasizes that more evidence is needed to demonstrate the validity of an examination than just its apparent fidelity to the job.
"Testing Across the
Nation" by Barbara ShowersShowers indicates that the need for examination security seems to be greater than ever. Her column describes some of the sophisticated devices now available to potential cheaters as well as steps credentialing agencies can take to improve test security in the face of these threats.
"The Answer Key"
by Norman HertzHertz's column raises and answers four questions. The first responds to a question concerning how to evaluate the quality of a criterion-referenced passing score study. The second concerns the implications for testing when the results of an occupational analysis indicate that activities not mentioned in the regulations are being performed by practitioners. The third question deals with the issues involved when a board considers using a performance test as part of its licensing process. The last question deals with concerns regarding test standards and whether they are consistent from state to state.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer identify and summarize a number of papers, articles, and reports reflecting current thinking on performance assessment, situations encountered in adapting performance assessments to licensing/certification applications, and historical perspectives of performance testing.
"Software Review"
by Lee Schroeder and Pansy Dubose HoughtonSchroeder and Houghton discuss how the use of computers in examination development processing, analysis of test results, and the storage of confidential information requires the design and implementation of tough security procedures in order to restrict access to appropriate personnel.
"The Death of PMPs and
the Lesson for Performance Assessment" by Lynn C. WebbWebb's column discusses the factors that caused the death of PMPs. She describes the psychometric difficulties as well as the financial and timing issues that resulted in the demise of this technique.
"Legal Perspectives of
Examination Security" by Dale AtkinsonAtkinson's article deals with the legal aspects of test security. The author discusses the ways in which the security of examinations can be breached. He then describes the procedures and legal actions boards and examination contractors can take to protect themselves and the public as well as discouraging individuals and organizations from attempting to breach test security.
"Licensing and
Certification Test Construction: A Balancing Act" by Wade Gibson and John
WeinerThis article describes the procedures that can be used to ensure that alternate forms of licensing and certification examinations meet both content and psychometric standards. The approach used is based on classical test theory and can be used with computerized tests to enable users to administer a unique alternate test form to each candidate.
"Testing Across the
Nation" by Barbara ShowersShowers covers four themes: the Americans With Disabilities Act and it focus on describing the "essential functions" of jobs; the National Board of Examiners' decision to stop using the latent-image gathering format in the Patient Management section of its certifying exam; the security concerns regarding computer-based tests; and the State Post-secondary Review Entity, which may require regulators to provide information on passing rates for institutions receiving Title IV funds that have educational programs leading to licensure.
"The Answer Key"
by Norman HertzHertz's column raises and answers five questions. The first concerns how examinations are affected when interpreters are used to translate an examination into a language other than the one in which it was written. The second asks what can be done to ensure that more items survive an item writing workshop. The third discusses the options a board has other than accepting passing scores that have been established by a national association. The fourth asks how to evaluate the quality of test questions. The last one is concerned with the meaning of validity in a licensing examination.
"The Americans with
Disabilities Act Versus Your Board's Duty to Investigate Applicants: The ADA is
Winning!" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L. BloomThis article discusses the Americans With Disabilities Act and licensing boards' duty to investigate applicants. It is clear that boards need to be very careful about the questions they ask. Useful advice is provided on the kinds of questions that are appropriate and can be defended.
"Abstracts and Updates"
by Charles Kunce and Mary SandiferKunce and Sandifer review: guidelines for computerized adaptive tests, an article describing procedures designed specifically for estimating the reading level of test questions, an article and paper describing the effects of changing answers on multiple-choice examinations, as well as two articles dealing with setting passing scores.
"Software Review"
by Lee Schroeder and Pansy Dubose HoughtonSchroeder and Houghton's column notes that bringing subject matter experts together to write and review examinations is one of the major expenses associated with an ongoing testing program. Their column describes test questions by computer and by telephone.
"America's
School-to-Work Initiative: What Role Should Credentialing Agencies Play? "
by Meredith Mullins and Donald Ross GreenMullins and Green discuss the current emphasis on improving the transition from school to work. They point out what they believe are some of the unique capabilities and experiences that credentialing agencies could contribute to the solution of this national problem. And they raise questions about the role of credentialing agencies in this area.
"Development of
Performance Assessments for Use in Professional Certification and Licensing"
by Janice Dowd ScheunemanScheuneman's article explores some of the difficulties that threaten the validity of performance measures. These include the lack of generalizability of scores and construct irrelevant variance present in scores.
"Psychometric
Evaluation of Performance Assessments: Critical Issues and Innovative Strategies"
by Richard M. JaegerJaeger's article describes the development of performance assessments by the National Board for Professional Teaching Standards (NBPTS). The NBPTS is developing performance assessments in a number of content areas that will be used to certify accomplished teachers. This article discusses the approaches used to develop the certification for Early Adolescence Generalists. It describes a variety of critical psychometric issues that arose in the development of these measures as well as the strategies used to address them.
"Testing Across the
Nation" by Barbara ShowersShowers discusses the use of computerized testing in licensing and certification. Barbara describes the various computer-based testing options, administration of computerized tests, impacts of computerized test delivery on the credentialing program, justifying the change, justifying the increased cost, availability of providers, components of cost, and the planning necessary for implementing computer-based testing.
"The Answer Key"
by Norman HertzHertz's column raises and answers five questions. The first deals with how to maintain the same level of scientific rigor in constructing the examination plan as was used in conducting and analyzing the job analysis. The second asks about the guidelines that can be used to define the entry-level practice when conducting a passing score study. The third deals with the factors to consider when developing procedures for scoring performance, problem, essay, or oral examinations. The fourth involves the role of board members in the examination development and scoring process; the fifth considers the role of educators in the examination development process.
"Unconstitutional
Advertising Restrictions" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L.
BloomThis article discusses the type of advertising restrictions that licensing agencies have adopted that have been upheld by the courts as well as the type of restrictions that may be unconsitutional.
"Abstracts and Updates"
by Mark RaymondRaymond's column discusses new publications in three major content categories. The first category is concerned with item writing. The articles in this section cover topics such as estimating the optimum number of options and the effects of altering the position of options in multiple-choice examinations as well as use of taxonomies in test construction. The second deals with procedures for use in estimating the reliability of performance tests. The third category includes articles on job analysis, testing policy, and computerized adaptive testing.
"Software Review"
by Lee Schroeder and Pansy Dubose HoughtonSchroeder and Houghton's column describes two software products they believe can very useful to regulatory agencies. One is Microsoft Access version 2.0, which is a database management tool. The other is DesignExpert, which enables an agency to produce its own scannable documents.
"Accommodating
Different Ethnic and Cultural Groups in Credentialing Examinations" by
Meredith Mullins and Donald Ross GreenMullins and Green discuss the importance of global awareness and sensitivity to cultural diversity in the design and administration of today's credentialing examinations.
"Evaluating Licensure
and Certification Examination Programs" by Steven M. Downing and Thomas M.
HaladynaDowning and Haladyna describe how high-stakes licensure and certification programs can benefit from external review. They discuss the advantages of such reviews, the aspects of the credentialing program that should be reviewed, the standards that should be considered for evaluating the program, as well as criteria to use when selecting an outside evaluator.
"Expanding a
Professional Assessment Program: Issues for Cross-Cultural Testing and Test
Translation" by Lynn C. WebbWebb's article notes that we have entered the age of the global community and that many certification and licensure programs are investigating the possibility of expansion. Lynn discusses issues and concerns related to cross-cultural testing and test translation. She emphasizes the need for planning, the use of appropriate testing standards, translation strategies, and special areas of inquiry.
"Testing Across the
Nation" by Barbara ShowersShowers discusses why she believes sole-source providers of credentialing examinations concerns about contract clauses, policy issues, and options for states. Barbara also discusses the progress being made on revising the Standards for Educational and Pyschological Testingpromulgated by the American Psychological Association, American Educational Research Association, and the National Council for Measurement in Education. Barbara also provides a list of organizations and publications that can be used as resources for issues related to performance-based assessment.
"The Answer Key"
by Norman HertzHertz's column raises and answers five questions. The first deals with whether licensing examination programs should provide diagnostic feedback to candidates who do not pass examinations. The second asks if a passing-score workshop is necessary to establish a criterion-referenced passing score for every examination--even if a substantial percentage of test questions are reused from administration to administration. The third concerns factors to consider when selecting subject-matter experts so as to ensure they represent practice. The fourth concerns issues related to the use of oral examinations; and the fifth deals with the appropriateness of subject-matter experts writing test questions in advance of item-development workshops.
"Licensure Denials:
Know Your Authority" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L.
BloomThis column discusses the importance of a board knowing its authority under its state's administrative procedures act when imposing disciplinary procedures.
"Abstracts and Updates"
by Mark RaymondRaymond's column discusses six articles concerned with the validation of licensing and certification examinations; six articles dealing with item-writing and test assembly; five articles concerned with alternative assessment; and three articles related to legal and administrative issues.
"Software Review"
by Lee Schroeder and Pansy Dubose HoughtonLee Schroeder's column applies his criteria for evaluating an item-banking system (published in a previous CERcolumn) to two item-banking systems that were sent to him for review. One of Lee's associates, Pansy Houghton, reviews LXR-Test 5.0 and The Question Bank.
"Gender Bias in
Licensure Testing: Is It a Problem" by Meredith Mullins and Donald Ross
GreenMullins and Green discuss the possibility of gender bias in licensure testing.
"Evaluating Test
Fairness in Licensure Testing: The Sensitivity Review Process" by Stephen
G. Sireci and Laura A. MullaneSireci and Mullane discuss the sensitivity review process as one aspect of evaluating test fairness in licensure testing. They describe the purpose of sensitivity review and the sensitivity review process, and also provide examples of problematic items as well as a sample rating sheet for documenting the results of a sensitivity review.
"Providing
Interpreters for Deaf and Hard of Hearing Candidates for Credentialing
Examinations" by Catherine NelsonNelson's article discusses issues related to providing interpreters for deaf and hard of hearing candidates for credentialing examinations. She outlines an interpreter's responsibilities, as well as some of the decisions a board needs to make when using interpreters. Catherine also identifies a source boards can use to identify qualified interpreters.
"Testing Across the
Nation" by Barbara ShowersShowers discusses five initiatives being undertaken to improve national standards, guidelines, and policy guidance for the developers and users of tests. Barbara also describes the results obtained from the National Adult Literacy Survey as well as business news about contractors working in the licensing and certification area.
"The Answer Key"
by Norman HertzHertz's column raises and answers four questions. The first concerns the amount of testing time to allow for candidates with learning disabilities. The second deals with how to report the scores for candidates with disabilities who have received testing accommodations, and the third discusses how to evaluate the quality of an occupational analysis that is conducted as a basis for test development. The fourth describes the issues that should be considered when exploring the option of changing from a paper-and-pencil test to a computer-administered version of the examination.
"Protecting Yourself
from Liability as a Board Member" by Susan E. Dorn, Kim A. Zeitlin, and
Margaret L. BloomThis column discusses the use of a "hold harmless clause" as one precaution that may reduce your board's potential exposure to a lawsuit.
"Abstracts and Updates"
by Mark RaymondRaymond's column discusses articles and books dealing with performance testing, job analysis and test specifications, and empirical validity in licensing and certification. Mark also mentions articles which describe techniques that can be used to set passing scores.
"Software Review"
by Lee SchroederLee Schroeder reviews the item banking system developed by Computer Adaptive Technologies, Inc.
"In Search of Truth
and the Perfect Standard-Setting Method: Is the Angoff Procedure the Best
Available for Credentialing?" by Meredith Mullins and Donald Ross GreenMullins and Green discuss whether the Angoff Procedure is the best standard-setting method available for use in the licensing and certification context.
"Legal Bases for
Licensure Testing" by William A. MehrensMehrens writes about the legal basis for licensure testing and discusses the related legal setting, professional standards, and state and federal court decisions.
"Computerized Adaptive
Testing for Licensure and Certification" by Betty A. Bergstrom and Richard
C. GershonNelson's article discusses issues related to providing interpreters for deaf and hard of hearing candidates for credentialing examinations. She outlines an interpreter's responsibilities, as well as some of the decisions a board needs to make when using interpreters. Catherine also identifies a source boards can use to identify qualified interpreters.
"Testing Across the
Nation" by Barbara ShowersShowers discusses the use of uniform national passing scores by veterinarians, psychologists, and physical therapists. She also describes a workshop on test disclosure that was cosponsored by CLEAR and NOCA, as well as two new publications distributed by CLEAR entitled "Development, administration, scoring, and reporting of credentialing examinations: Recommendations for board members" and "Principals of fairness: An examining guide for credentialing boards".
"The Answer Key"
by Eric WernerWerner's column raises and answers two questions. The first deals with the pros and cons of constructed response items and the second with how to go about setting a passing score for a performance examination.
"Reviewing Your Exam
Review Procedures: How Well Do You Score?" by Susan E. Dorn and Kim A.
ZeitlinDorn and Zeitlin discuss several cases involving challenges to exam review procedures. They provide some useful information concerning review procedures for multiple-choice and essay type examinations.
"Abstracts and Updates"
by Mark RaymondRaymond's column reviews two new books and seven articles concerning computerized adaptive testing and item response theory, setting and maintaining standards over different forms of an examination, the legal defensibility of high-stakes tests, and measurement problems.
"Software Review"
by Lee SchroederLee Schroeder describes the important features he believes an item banking system should possess and the criteria he feels should be used to evaluate such systems.
"The Beuk Compromise
Adjustment: Possible Rx for Troubled Cut Score Study Results" by F. Jay
BreyerBreyer discusses a method for adjusting passing scores that may be helpful to credentialing boards that find themselves faced with recommendations for passing scores that appear to be either too strict or too lenient. The article describes how the Beuk Compromise Adjustment can be used with the Angoff technique to improve decision-making about where to set the passing score.
"Content Validity
Procedures" by I. Leon Smith and Sandra GreenbergSmith and Greenberg describe some approaches that can be used to strengthen the documentation and demonstration of content validity. The authors describe several methods for evaluating the relationship of items to test specifications. These procedures, when implemented in conjunction with those used to develop the examination, can strengthen the documentation of content validity.
"Testing Across the
Nation" by Barbara ShowersShowers describes the changes taking place in several key national licensing examinations. These examinations are for accountants, architects, medical doctors, and nurses. The changes include: permitting and controlling the use of calculators, assessing writing skills, and using computers to administer and score examinations.
"The Answer Key"
by Eric WernerWerner's column raises and answers three questions. The first is about the need and desirability of translating licensing examinations into languages other than English, the next about how a board can decide whether or not to change their closed-book examination to an open-book assessment, and the third concerns how much information about a test a licensing board should provide candidates.
"Test Score Reporting:
What Is Your Discretion?" by Susan E. Dorn and Kim A. ZeitlinDorn and Zeitlin discuss several cases dealing with a testing agency which refused to report a candidate's score because of doubts about the validity of that score. Despite numerous cases approving this policy, this course of action is now under attack.
"Abstracts and Updates"
by Mark RaymondRaymond's column reviews three sets of articles. One deals with a variety of multiple-choice formats. He then discusses articles on other types of testing formats, such as oral tests and constructed response tests. This is followed by a presentation of general articles on validity.
"Software Review"
by Lee SchroederLee Schroeder describes and reviews an item analysis program, ITEMSTAT, which was developed by the Colorado Department of Regulatory Agencies. Lee discusses how this program can be used to identify questionable items.
"Assessing Clinical
Skills in Optometry: A National Standardized Performance Test" by Leon J.
GrossGross describes the performance test used as part of the licensing process for optometrists. The article describes the examination; the training given the examiners; the Candidate Guide; and the scoring procedures. This article provides a good overview of a number of complex technical and administrative issues that must be resolved when conducting performance tests in a licensing context.
"Standard Setting in
Compensatory Versus Noncompensatory Licensure Testing Programs" by Neal
KingstonKingston discusses standard-setting in compensatory and noncompensatory licensure testing programs and its impact on passing rates.
"Testing Across the
Nation" by Kara SchmittSchmitt presents the second of her two-part series describing the key issues that licensing and certification professionals believe will most affect licensure and certification in the next three to five years. Kara discusses the following topics: the need for better procedures and documentation to ensure that the content of examinations is job-related and that passing scores truly reflect appropriate standards; the need to provide evidence that licensing and certification really do play a role in protecting the public; continuing education; and the possible involvement of the Federal government in licensure.
"The Answer Key"
by Eric WernerWerner's column raises and answers three questions. The first is about the need for, or advisability of, providing a grade-level readability index for multiple-choice tests used in licensure, the next considers whether English as a second, nonprimary language, constitutes a disability, and the third concerns ways in which a licensing or certification agency can develop valid and fair oral examinations.
"State Licensing Board
Alert: Keeping Communications Confidential" by Susan E. Dorn and Kim A.
ZeitlinDorn and Zeitlin discuss whether a state licensing board's investigatory files are discoverable. They also provide a checklist which may be helpful to a board in dealing with such matters.
"Abstracts and Updates"
by Mark RaymondRaymond's column alerts us to four new books; one deals with continuing education, tow with computer-adaptive testing, and a fourth with generalizability theory as a tool for evaluating the reliability of scores for virtually all types of performance tests. A number of articles are also presented that cover topics such as: computer-adaptive testing, an independent auditing mechanism for testing, item-banking, and the validity of job analysis data.
"Software Review"
by Lee SchroederLee Schroeder describes and reviews a software package used by one of our readers. The product is the Test Development and Analysis System (TDAS) published by Applied Psychometric Services, Inc. of Napierville, Illinois. Lee discusses installing TDAS as well as its capacity to process tests and to provide reports.
"The Validity of
Licensing and Certification Exams" by Benjamin ShimbergShimberg's column deals with the validity of licensing and certification examinations. Ben discusses professional standards, how licensing and certification tests differ from those used for selection, content validity, and the pros and cons of construct validity.
"Psychometric Issues
in the Use of Simulations and Work Samples as Examinations" by Ellen R.
Julian and Nancy A. OrrThis article should be useful to raders who are considering or currently using performance tests as part of their licensing or certification process. Ellen and Nancy discuss some of the difficult psychometric issues that still need to be resolved when using measures of this type.
"Evaluating Items for
Fairness" by Michael ZiekyZieky's article describes a multi-state study that was conducted to evaluate the fairness of multiple-choice questions. The author presents the definition of unfairness that was used, the reactions of the reviewers, and aspects of questions unrelated to fairness that reviewers tended to confuse with fairness issues.
"Testing Across the
Nation" by Kara SchmittSchmitt focuses on the results of a survey she conducted recently of professionals working in the area of licensing and certification testing. Kara asked these individuals to identify the key issues they felt would most affect licensure and/or certification in the next three to five years. Her column is the first of a two-part series describing her findings. She discusses the impact of computerization, the use of practical examinations, privatization, and the nontypical licensure candidate.
"The Answer Key"
by Eric WernerWerner's column raises and answers questions about four interesting issues. The first is a policy question that deals with the appropriateness of allowing candidates an unlimited number of test retakes, the next concerns which reference group to use when examining the passing rate of a test, the third considers how a board might obtain assistance to improve the quality of its testing services and to negotiate better with their test service provider, and the fourth concerns the appropriateness of a board providing candidates' test scores to employers for use in making personnel decisions.
"Legal Issues" by
Susan E. Dorn and Kim A. ZeitlinDorn and Zeitlin discuss two developments which may help us understand how the courts may apply the new Americans with Disabilities Act to licensing and certification. There is also a brief discussion of a very recent case dealing with a Florida title statute.
"Abstracts and Updates"
by Mark RaymondRaymond's column identifies two sets of articles. The first set discusses the latest trends in standard setting. The second set presents a debate over the limitations of multiple-choice questions as well as some of the new "authentic" assessment methods.
"Software Review"
by Lee SchroederLee Schroeder reviews his experience with Paradax, a relational data base, and Bubble Publishing. As you may recall from Lee's previous column, a relational data base is a collection of linkable files. Bubble Publishing allows the users to design two types of optically readable answer sheets.
"Psychometric Issues"
by Benjamin ShimbergShimberg's column contains an article by Ronald Hambleton that provides background on item response theory and why it is expected to play a greater role in licensing and certification in the 1990s.
"Measurement Practices
in Licensing Examination Programs: A Survey" Stephen J. Sireci and Bruce H.
BiskinThis article reports on a survey of measurement practices in national licensing programs. The article describes the approaches being used to assess competency, set passing scores, and document evidence of reliability and validity.
"Computerized
Testing--Mind Your Marketing" by David ValeVale's article describes some of the advantages and disadvantages of using computerized testing. It provides some useful hints for determining whether or not computerized testing would be appropriate for your assessment program.
"Testing Across the
Nation" by Kara SchmittSchmitt discusses questions raised by the Americans with Disabilities Act (ADA), legal challenges affecting examinations, computer-administered examinations in Wisconsin, a patient consultation examination, and self-assessment.
"The Answer Key"
by Eric WernerWerner's column raises and answers questions about statistical procedures for use in identifying whether one candidate taking a multiple-choice examination might have copied from another and how item analysis might differ for a multiple-choice and a performance test.
"Legal Issues" by
Susan E. Dorn and Kim A. ZeitlinDorn and Zeitlin discuss the accommodations required for disabled candidates by the Americans with Disabilities Act that are relevant for licensing and certification agencies. These accommodations include access to exam administration facilities as well as certain types of assistance in taking the exam.
"Abstracts and Updates"
by Mark RaymondRaymond's column covers legal and technical issues related to validity, item bias, and alternative testing methods.
"Software Review"
by Lee SchroederSchroeder discusses the value of laptop of computing, the use of a relational data base system, and the establishment of a CLEAR software clearinghouse.
"Psychometric Issues"
by Benjamin ShimbergShimberg's column explains what equating does and discusses how it contributes to test fairness.
"On the Role of
Criterion-related Validity Evidence for Licensure Examinations" by Michael
T. KaneKane's article examines the use of a criterion-related validity for licensing examinations.
"Continuing Competency"
by Joan E. KnappKnapp discusses the issue of continued competency. While the article does not solve the problems associated with continuing competency, it will give you a better understanding of the nature of the difficulties associated with this topic.
"Testing Across the
Nation" by Kara SchmittSchmitt discusses a new commission reporting on testing, a new appraiser credentialing program, and new challenges for special test accommodations.
"The Answer Key"
by Eric WernerWerner's column answers a specific question on peridodic updating of job analyses.
"Legal Issues" by
Susan E. Dorn and Kim A. ZeitlinDorn and Zeitlin discuss a recent Supreme Court decision regarding a state bar challenge to an attorney for noting on stationery letterhead that he held specialty certification. This decision has substantial implications for professionals and regulators.
"Abstracts and Updates"
by Mark RaymondRaymond's column provides a comprehensive update on recent publications on job analysis.
"Software Review"
by Steve NettlesNettles reviews some software utilities to enhance computer productivity.
"Psychometric Issues"
by Benjamin Shimberg and Michael RosenfeldShimberg and Rosenfeld discuss job analysis methodology. They discuss how a job analysis is conducted, a committee's judgements are verified, and the results are used.
"Adapting the Code of
Fair Testing Practices in Education to Licensure Testing" bu Donald Ross
GreenThis article is a recommendation to extend the Code of Fair Testing Practices in Education to licensure and certification tests. What a great idea! Although credentialing examinations are given in the public interest, these are also "high stakes" tests for the candidates. Therefore, providing assurances of fairness to test takers is appropriate and desirable.
"Norm- vs.
Criterion-Referenced Passing Scores: Considerations for Passing Rates" by
John MironeMirone discusses the important distinctions between norm- and criterion-referenced standard setting procedures. This article is neither the first nor the last to be written on this subject. However, the author provides a somewhat different perspective on the issues, one which will influence our thinking.
"Testing Across the
Nation" by Kara SchmittSchmitt discusses a a significant legal decision, a realignment of major exam programs, and the effects of exam preparation.
"The Answer Key"
by Eric WernerWerner's column provides insightful answers to questions that are frequently asked. Topics include multiple pass-fail cutoff scores, pretesting new items, professional standards, and test equating.
"Legal Issues" by
Kim A. ZeitlinZeitlin discusses a recent setback to truth-in-testing laws due to a federal district judge's decision. The article includes observations about copyrights which were declared supreme by the ruling.
"Abstracts and Updates"
by Mark RaymondRaymond's column provides some excellent summaries of recent publications on an important topic--test item writing.
"Software Review"
by Steve NettlesNettles reviews an inexpensive statistical package.
"Psychometric Issues"
by Benjamin ShimbergShimberg's column delves into important psychometric issues or statistical indices. In this issue, he discusses practical examinations or performance tests. The ideas raised will resolve some commonly held concerns.
"Testing Candidates
with Disabilities" by Catherine Nelson and Marjorie RagostaThis article discusses linguistic issues from the perspective of disabled or handicapped individuals. For whom should the test administration format be adjusted? What should be the extent of the adjustment?
"Comparability of
Translated Tests in Occupational Testing" by Nancy Thomas AhluwaliaAhluwalia discusses issues involved in translating examinations. When and how should exams be translated, and what problems may be expected?
"Meeting the Needs of
Candidates With Limited Literacy" by Charles B. Friedman and Barbara HalseyFriedman and Halsey discuss linguistic issues from a very different perspective. Their concern is for licensure exams, when the literacy level required to take the test far exceeds the level required on the job. How should this be handled?
