CLEAR Exam Review
Contents of Journal Issues Beginning with Summer 1990
All back journal issues are available for $15 each. There is an additional $3.75 charge for shipping and handling, plus $1 for each additional item

George Gray's column cites a number of chapters in a new book "Improving Testing: Applying Process Tools and Techniques to Assure Quality" as well as discussing a number of articles dealing with the following topics: practice analysis, standard setting, retest effects on same and parallel test forms, testing with dual languages and audio, accommodations for testing, and equating.

Robert Shaw deals with the topic of biometric security systems. He describes common authentication methods, typical biometric configurations, characteristics of biometrics, and biometric system errors. This is the first of a two-part article dealing with biometric security systems.

Dale Atkinson's legal column discusses a case involving a situation in which a licensing board produced a practical examination in a haphazard manner. The court ruled against the licensing board. This case clearly demonstrates that part of the defensibility of an examination relies on the fundamental process used to develop the assessment instrument.

This article deals with the importance of being able to identify item parameter drift in multistage adaptive tests. The authors also discuss various techniques for determining if item parameter drift has occurred.

This article discusses the item characteristics of several different innovative item formats.

George Gray's column features the fourth edition of Educational Measurement. He discusses chapters dealing with the state of the field in educational measurement, questions for future standard setting research, quality control measures in testing, evidence-centered design and testing, and multistage computer-based testing.

Robert Shaw and Robert Clark discuss the use of digital images as test item stimuli. They note the possible advantage of using true-to-life stimuli in certification examinations as a way to improve the relevance and validity of the examination. They discuss the challenges associated with the use of visual images, some possible solutions to those challenges, and provide a set of recommendations for producing acceptable images on screen as well as in a paper format.

Dale Atkinson's legal column notes that regulatory boards are created and empowered to regulate professions and in doing so may consider prerequisites such as education, experience, and examinations. He further notes that in some professions and jurisdictions additional criteria such as good moral character and other personal history variables are considered. Dale discusses a case involving the number of times a candidate is allowed to take an examination and how that requirement changed over time. This case presents many issues for consideration related to the important public protection mission of regulatory boards. These include the legal authority to limit the number of examination attempts allowed, the order in which licensure requirements are assessed, as well as other relevant issues.

This article describes a study designed to focus on the activities performed by post entry-level Registered Nurses. The procedures used and results of this study may be useful to others considering methods for assessing or assuring the continued competence of credentialed professionals.

George Gray cites articles dealing with the following topics: validity in court, point counter-point regarding task-analysis methodology, point counter-point regarding standard-setting methodology, additional standard setting articles, item response theory, and differential item functioning.

Robert Shaw deals with the topic of web conferencing.  He describes the primary function of a web conference, the features available, costs, security, and human factor issues.  The column provides a good deal of useful information for those of you considering web conferencing.

Dale Atkinson's legal column discusses a case involving a test preparation company and the procedures they used to make their practice examinations look like the actual licensing examination.  These procedures included having members of the test preparation company take the licensing examination multiple times and write down questions on "scratch paper."  The court ruled that the test preparation company had violated copyright laws and ordered that they pay a substantial fine.

This article deals with using web-based virtual meetings to make the test development process more efficient and cost-effective.  It discusses the types and requirements for virtual meetings as well as their advantages and disadvantages.

This article describes the use of talk-aloud protocols to explore the kinds of cognitive functioning examinees - recently licensed people - use to answer both multiple-choice and alternate item types.  The authors provide some interesting results and conclude that talk-aloud studies such as those described are feasible and provide further evidence of what certain items are measuring.

George Gray discusses several of the chapters in the new Handbook of Test Development. Several of the chapters are reviewed and discussed including some discrepancies among the chapter authors. He also reviews an article on computer adaptive testing that provides an overview of the test delivery methodology. Finally he discusses an article describing a literature review of the Bookmark Standard Setting Method.

Robert Shaw deals with the topic of self-service systems in general and specifically their use in credentialing programs. An example of a self-service system is an on-line registration system.

This article describes a study comparing paper-based practice analysis surveys delivered through the mail and an Internet-delivered survey. The results are interesting and deserve your attention.

This article provides an overview of the potential of computerized test delivery for making classification decisions. The article describes how a computerized classification test is developed and its potential utility and limitations.

George Gray cites a number of articles, measurement books, and presentations dealing with the topics of score interpretations, a survey of measurement, equating issues, changing answers on multiple-choice examinations, the case for three-option multiple-choice questions, and a discussion of a variety of issues related to computer adaptive testing.

Lee Schroeder and Reed Castle deal with the topic of reporting scores.  They provide a survey of comments on the different ways credentialing programs report scores in light of validity considerations.

Dale Atkinson's legal column discusses an interesting issue related to state licensing rights versus due process rights.  The regulated profession discussed is retail florists and raises some interesting perspectives regarding protection of the public.  The column describes the review and conclusion of the court.

This article describes the use of Differential Item Functioning (DIF) to examine the fairness of knowledge-based licensure examinations administered to candidates in nursing who use English as a Second Language (ESL).  Their approach to answering questions of test fairness as a result of primary language through DIF analyses as well as their results makes this article worth your attention.

This article describes the use of an internet-based survey methodology to efficiently conduct a job analysis for a widely dispersed group of professionals - in this case Naval Architects and Marine Engineers.  His perspective and commentary on how the process worked for his profession is interesting and we think provocative in some respects.

George Gray cites a number of articles dealing with topics of interest: using skill, knowledge, and ability statements to develop a weighted content outline; flagging scores from accommodated test administrations; setting performance standards; formula scoring and key balancing; rating a subset of items for an Angoff study; reporting examination subscores; weighting examination components; and constructing measures using item response theory (IRT).

Lee Schroeder and Reed Castle discuss the delivery of an examination from a CBT network server to the candidate workstation.  Two general types of CBT delivery modes are discussed: a Local Area Network-based Administration and an Internet Administration.  The benefits and limitations of each type are discussed.

Dale Atkinson's legal column discusses an interesting case involving a decision to invalidate test scores.  He discusses the issue of who makes the ultimate determination as to score invalidation and the consequences of such a decision.  In the case cited, the court recognized the importance of deference to testing experts and the need to allow for group invalidation of scores in order to maintain the integrity of the entire examination program.

This article deals with the emergence of Internet sites in which candidates post items from live credentialing examinations.  The article describes an experiment designed to evaluate the impact of posted items on the difficulty and validity of a certification examination over time.

This article discusses the equating of licensing and certification examinations.  The article describes the important factors credentialing agencies must consider when choosing and implementing an equating process.  Recommendations for dealing with these and other issues are provided.

Sandy Greenberg deals with the challenge of testing the repeating licensure candidate.  She discusses the issue from a policy and legal point of view.

George Gray cites a number of articles dealing with the topics of multiple choice item development, item response theory (IRT) item analysis, IRT model fit and application to standard setting, adapting tests to foreign languages, computer-based testing issues, technical issues in score equating, fairness review guidelines, and IT certificatiot>

"Technology and Testing" by Lee Schroeder and Reed Castle

CLEAR Exam Review, Summer 2004, Volume 15, No. 2

CLEAR Exam Review, Summer 2003, Volume 14, No. 2

CLEAR Exam Review, Winter 2003, Volume 14, No. 1

Greenberg discusses two major topics: converting the Uniform CPA Examination to a computerized administration and using automated test assembly to construct multiple test forms for online administration.

Atkinson discusses a case that involves the forfeiture of a medical license in a Federal criminal case. The column focuses on the license as personal property that can be used in federal sentences without regard to the state rules regarding revocation.

CLEAR Exam Review, Summer 2000, Volume 11, No. 2

  "Testing Across the Nation" by Sandra Greenberg and Karen Cullen

Greenberg and Cullen focus on innovative uses of technology in credentialing programs. They discuss the use of electronic data in a practice analysis study, Web-based self-assessment examinations, online application processing, Web access to credential information, and online license renewals. 

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss recent articles, abstracts and presentations on differential item functioning (DIF), performance assessment, and computers and testing.

"Software Review" by Lee Schroeder

Schroeder's reviews IMSI's HiJaak Pro Version 5, a software product that allows the user to convert, capture and organize graphic files. This should mean that graphics can be stored and used more easily in test questions. 

"Legal Beat" by Dale Atkinson

Atkinson discusses a recent case that focuses on the appropriateness of testing agencies and organizations disclosing to recipients of test scores that the examination was administered under nonstandard conditions. The opinion in the case analyzed Title III of the Americans with Disabilities Act (ADA) as it relates to private entities providing examinations to the public.

"Testing and Measurement Issues" by Thomas Henzel and Kristina Golden

Henzel and Golden focus un the the structural complexity of test items for computer-based testing. Their column identifies a new classification code that describes a test item in terms of its structural complexity. They suggest this new code may be useful in the reproduction of multiple examination forms that are balanced for content and difficulty, as well as complexity of item construction.

"Defining the Scope of a New Job and its Knowledge Base: The Application of a Job Analysis Study to
        a Decision-Making Process" Michael Rosenfeld, Sarah Slater and Sharon Goldsmith

The authors describe how job analysis techniques may be used to define the scope of a new job, assist in defining the curriculum necessary to perform the tasks and acquire the skills necessary for competent entry-level performance, and serve as a basis for standards to be used to credential approved training programs.

"Condensed Job Analysis: Capturing a Moving Target in the Information Technology Field" by James
        Adair

Adair discusses the development of an alternative job analysis method. It focuses particularly on the adaptation of existing job analysis techniques and processes to meet the needs of the Information Technology industry for assessing quickly altering job and product-related content. 

CLEAR Exam Review, Winter 2000, Volume 11, No. 1

  "Testing Across the Nation" by Jim Zukowski

Zukowski discusses an audit program recently approved in the state of California that will impact upon credentialing examination programs related to professional licensure. He also reviews recent U.S. Supreme Court decisions that will affect the application of ADA with regards to testing accommodation requests, provides an update on a job analysis for social workers, and describes a new certification program for information technology specialists.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss recent articles, abstracts and presentations on test adaptation, computer-based testing, and the impact of new technology on examinations. Other issues include gender differences, validation and performance testing and the accuracy of item ratings of subject matter experts.

"Software Review" by Lee Schroeder

Schroeder's reviews the Statsoft Electronic Statistics Textbook, a book of reference information that can be accessed for free on the Internet. He recommends this particularly for those that find themselves in need of information on statistics when out of the office.

"Legal Beat" by Dale Atkinson

Atkinson discusses a case of a dentist with a limited license from the state of Arizona. After the legislature repealed the section of legislation providing for limited licenses, the dentist filed a complaint for violation of his due process and equal protection rights.  

"Testing and Measurement Issues" by Gerard Dillon and William Walsh

Dillon and Walsh focus on using performance data to set standards, its practical impact and the perception of judges. They also summarize the results from a research study that they recently conducted.

"On the Documentation of Credentialing Examination Procedures and Policies" by James Fidler

The author considers issues pertinent to the preparation of performance and policies manuals for credentialing examination programs. The issues discussed by Fidler include: the importance and uses of documentation; contents and format of a manual; appropriate author and audience; and document storage.

"A Brief Discussion of Three Basic Computer-Based Testing Models" by Jerry Gorham and Jian Zhang

Gorham and Zhang provide an introduction to computer-based testing and discuss the different models available. 

CLEAR Exam Review, Summer 1999, Volume 10, No. 2

  "Testing Across the Nation" by Jim Zukowski

Zukowski discusses a new partnership between the Chauncey Group and the NorthWest Center for Emerging Technology (NWCET) concerned with the Information Technology industry. Also mentioned are the revised Standards for Educational and Psychological Testing, an update on the practice analysis in orthotics and prosthetics, computer-based podiatry testing, the development of a set of guidelines for computer-based testing, and new standards for food protection managers.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss recent articles, abstracts and presentations on educational standards, computer-based testing (item exposure, automated test assembly, and user interface),  ADA accommodations, performance assessment, equating and scaling, and job analysis. 

"Software Review" by Lee Schroeder

Schroeder's article focuses on Item Response Theory (IRT) and provides a review of the theory, before considering a software product (XCALIBRE) which can be used to calculate IRT parameters.

"Legal Beat" by Dale Atkinson

Atkinson discusses a recent Missouri case that focused on the reliance of regulatory boards on licensure examinations and whether limitations to this authority exist. 

"Testing and Measurement Issues" by Brian Klauser and Ronald Nungester

Klauser and Nungester focus on the factors that need to be considered when identifying cut scores in certification and licensure decisions.

"Adapting Credentialing Exams for Use in Multiple Languages" by Ronald Hambleton, Stephen Sireci,
        and Frederic Robin

The authors consider reasons for adapting examinations for use in other languages and steps that can be taken to adapt the examinations. Eight steps are recommended: checking exam content and format equivalence; deciding the desirability of a translation; choosing translators; translating and adapting the exam; reviewing the adapted version of the exam; conducting a small tryout of the adapted exam; carrying out a more ambitious study of the adapted exam; and documenting the process.

CLEAR Exam Review, Winter 1999, Volume 10, No. 1

  "Testing Across the Nation" by Jim Zukowski

Zukowski discusses a number of topics that are currently being considered by several state and national boards. His column discusses such issues as the transition to computer-based testing, examination ownership, and multi-state job analysis.

  "Testing, Testing...1, 2, 3" by Grady Barnhill and Lynn Webb

Barnhill and Webb talk about credentialing agency responsibilities to the failing candidate. They discuss the pros and cons of providing diagnostic feedback to candidates.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss the status of both the revised Standards for Educational and Psychological Testing and the "Rights and Responsibilities of Test Takers". Additionally they review articles dealing with the detection of cheating, consequential validity, item relevance, learning disabilities, standard setting, and essay and performance test scoring.

"Software Review" by Lee Schroeder

Schroeder discusses concerns about the security of documents that are sent using e-mail or through the Internet. Lee describes software that uses public key cryptography to help solve this problem.

"Legal Beat" by Dale Atkinson

Atkinson discusses how managed care can make issues of defining scope of practice and determining jurisdiction quite challenging to regulatory boards. 

"Testing and Measurement Issues" by Shelby Keiser

Keiser discusses factors licensing boards should consider when responding to individuals requesting test accommodations. She emphasizes the fact that licensing agencies are not educational institutions. Their mission is public protection.

"Security for CBT High Stakes Licensure Exams" by Barbara Halsey

Halsey discusses the way in which the National Council of State Boards of Nursing attempts to maintain the security of its computer administered licensing examinations. These include such aspects as data transmission and back-up, software security, the training and certification of administrators/proctors, identification/check-in procedures, confidentiality agreements, and the testing environment.

"Guidelines for Selecting a Standard Setting Panel for Licensure Testing" by David M. Williamson

Williamson discusses factors to consider when selecting participants in a standard setting panel. He emphasizes that these factors are important in order to ensure the integrity of the standard setting process.

CLEAR Exam Review, Summer 1998, Volume 9, No. 2

  "Testing Across the Nation" by Jim Zukowski

Zukowski discusses methods related to standard setting for performance assessments and describes a new method developed by the Education Commission for Foreign Medical Graduates.

   "Testing, Testing...1, 2, 3" by Grady Barnhill and Lynn Webb

Barnhill and Webb talk about all the things you need to consider when facilitating meetings. They discuss how to encourage subject matter experts to attend meetings, the ideal group size, and the role of the facilitator.

  "Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss articles dealing with political and regulatory issues, performance testing, and standard setting methods.

  "Software Review" by Lee Schroeder

Schroeder talks about a problem of concern to many of us - the security of our computers. Lee discusses software that is available to protect the security of your files in the event your laptop is lost or stolen.

  "Legal Beat" by Dale Atkinson

Atkinson discusses one of the major functions of legislatively created regulatory boards - the conduct of disciplinary proceedings against licensees accused of wrongdoing. He discusses the standards of proof necessary for a board to establish wrongdoing of an accused licensee.

  "Testing and Measurement Issues" by Richard Luecht

Luecht discusses how moving from paper-and-pencil testing to computerized testing can require fundamental changes in the way tests are produced. He describes an automated test assembly approach that helps meet increased item development needs.

  "Job Analysis for High-stakes Credentialing Examinations" by Vicki L. Flaherty and James B. Hogan

Flaherty and Hogan discuss the role job analysis plays in test development and validity in credentialing examinations. They discuss a number of practical issues concerning how to plan, conduct, analyze, and report on the results of job analysis studies.

  "The National Board of Podiatric Medical Examiners New Testing Methodology" by Charles Gibley,
         Jr.

Gibley describes how the testing program conducted by the National Board of Podiatric Medical Examiners evolved over several years from a paper-and-pencil program to one that is based on computerized mastery testing (CMT). The article describes the steps that were taken to move from paper-and pencil tests to linear computer-delivered tests, and then on to CMTs.

CLEAR Exam Review, Winter 1998, Volume 9, No. 1

  "Testing Across the Nation" by Jim Zukowski

Zukowski discusses the "Guidelines for Documentation of a Learning Disability in Adolescents and Adults," developed by the Association on Higher Education and Disability (AHEAD). In addition to their use in higher education, these guidelines have important implications for candidates who are are applying for test accommodations for licensing and certification examinations. Zukowski also describes the procedures used by the National Institute for Hearing Instrument Studies to develop their international licensing examination for hearing instrument dispenser licensing boards.

  "Testing, Testing...1, 2, 3" by Grady Barnhill and Lynn Webb

Barnhill and Webb talk about how the delivery modes for assessing professional competency--ranging from oral exams to virtual reality testing--have changed over time. Their timeline extends from 2357 B.C. to the present. The conversation makes clear that some delivery modes are better geared to the achievement of particular assessment objectives than others.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer discuss articles dealing with different language and cross-cultural testing, bias and differential item functioning, significance testing, consequential validity, innovative uses of computers in testing, and performance testings.

"Software Review" by Lee Schroeder

Schroeder reviews a relatively inexpensive statistical package that he found to be very comprehensive and useful.

"Legal Beat" by Dale Atkinson

Atkinson looks at two opposing judicial opinions on the same issue to remind board members that judicial opinions may differ on the same issue, not only from state to state, but also from district court to district court, or from circuit court to circuit court. These differing opinions make a board member's job difficult and challenging.

"Testing and Measurement Issues" by Gerard Dillon

Dillon describes the use of survey data in a testing program. He discusses how asking test takers focused questions about their testing experience can provide useful information for modifying and improving various aspects of a testing program.

"Advances in the Use of Item Response Theory in Licensure and Certification Assessment" by Betty
      A. Bergstrom and Richard C. Gershon

Bergstrom and Gershon outline the differences between classical test theory and item response theory (IRT) and describe how applications of IRT are relevant to regulatory and credentialing organizations. They discuss the use of IRT for item banking, automated test construction, online test construction, computerized adaptive testing, computer-based simulations, performance assessment and surveys.

"Mental Model Comparison of Automated and Human Holistic Scoring of Architectural Design
         Simulations" by David M. Williamson, Issac I. Bejar and Anne Hone

The authors compare automated scoring with human scoring. Using data gathered from the Architect Registration Exam (ARE), they compare the computer-based scoring of open-ended architectural problems with the results obtained from human scorers.

CLEAR Exam Review, Summer 1997, Volume 8, No. 2

"Testing Across the Nation" by Jim Zukowski

Zukowski describes the new Multistate Pharmacy Jurisprudence Examination, a take-home recertification examination developed by the American Board of Internal medicine. He also examines the Internet as a way to provide information about licensing to candidates and the public.

"Testing, Testing...1, 2, 3" by Grady Barnhill and Lynn Webb

Barnhill and Webb provide useful information about assessing continuing competence in a lively, conversational manner.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer review a new book published by CLEAR, Demystifying Occupational and Professional Regulation: Answers to Questions You May Have Been Afraid to Ask, by Kara Schmitt and Ben Shimberg. Next, they cover a special credentialing issue of the journal Applied Measurement in Education, followed by a list of sessions at the 1997 meetings of the American Educational Research Association and the National Council on Measurement in Education that dealt with issues important to credentialing agencies.

"Software Review" by Lee Schroeder and Pansy Houghton

Schroeder and Houghton discuss a product that allows for in-house design and printing of double-sided, scannable forms.

"Legal Beat" by Dale Atkinson

Atkinson discusses two cases involving a board's authority to investigate the activities of licensees. In these two cases, the board's authority was upheld--but these processes may not be protected in all jurisdictions.

"Testing and Measurement Issues" by Anthony LaDuca

LaDuca describes the approach the National Board of Medical Examiners is taking to evaluate practicing physicians' continued competence, a topic of interest for many professions.

"New Testing Methodologies for the Architect Registration Examination" by Jeffrey F. Kenney

Kenney explains new testing methodologies being used by the National Council of Architectural Registration Boards. In particular, the article describes the development of procedures that enable a computer to score and administer small, focused design problems, known as "vignettes," which simulate an architect's design work.

"Is the Angoff Method Really Fundamentally Flawed?" by Michael Zieky

Zieky asks whether or not the Angoff method used to set passing scores is fundamentally flawed, as recently alleged. This article is particularly useful for those who use this standard-setting procedure.

CLEAR Exam Review, Winter 1997, Volume 8, No. 1

"Testing Across the Nation" by Barbara Showers

Showers makes a plea for the provision of diagnostic information to candidates who fail credentialing examinations. She discusses the psychometric considerations as well as some creative solutions that can be used to provide candidates some of the information they seek.

"The Answer Key" by Norman Hertz

Hertz responds to four common examination questions. The first deals with the ideal number and type of references upon which to base an examination. The second concerns possible problems associated with a state adopting an examination developed and administered by an association of state boards. The third responds to a concern raised by a small licensing program that wants to ensure its assessments comply with national standards. The fourth deals with the grounds on which appeals of examination results should be considered.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer review a new book on computer-based testing, six references dealing with general certification issues, three references concerned with the Americans with Disabilities Act, as well as six articles devoted to issues and viewpoints pertaining to performance assessment.

"Software Review" by Lee Schroeder and Pansy Houghton

Schroeder and Houghton discuss software that can be used to manage bands of examination questions. Their focus is on software that can be used to ensure that duplicate or very similar items are not unknowingly included in the item bank.

"Legal Beat" by Dale Atkinson

Atkinson discusses whether or not the exam scores of licensees should be made available to the general public.

"Testing and Measurement Issues" by Janice Scheuneman

Scheuneman provides an objective and balanced discussion of the advantages and disadvantages of computer-based testing.

"Using Nominal Group Technique (NGT) to Identify Factors Potentially Influencing a Consistently Low
         Passing Rate" by Linda M. Dean

Dean describes the use of the Nominal Group Technique to identify factors potentially influencing a consistently low passing rate. It incorporates a process borrowed from the research on group dynamics to solve a testing problem encountered in licensure.

CLEAR Exam Review, Summer 1996, Volume 7, No. 2

"Testing Across the Nation" by Barbara Showers

Showers explores whether or not credential test providers are performing reliability analyses at the cut score to determine which, if any, of the analyses have become "state of the art" among providers. The data for this column were collected via a survey sent to credential test providers.

"The Answer Key" by Norman Hertz

Hertz answers four common examination questions. The first relates to the advisability of asking item writers to prepare questions in advance of the item writing workshop. The second deals with how licensing and certification tests differ from academic (achievement) and employment examinations. The third concerns whether the completion of an occupational analysis and the setting of test specifications are sufficient to meet validity requirements for an examination used in licensing and certification. The fourth deals with how to explain passing score variation from one administration to another to candidates when using a criterion-referenced methodology.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer describe a new reference book on licensure testing, five articles discussing the use of computers to grade performance tests, an article on rater accuracy in performance testing, three articles about guidelines for translating exams, and two articles about candidate reactions to new licensure tests.

"Software Review" by Lee Schroeder and Pansy Houghton

Schroeder and Houghton discuss the strengths and weaknesses of introducing technology to reduce or even substitute for the need to have face-to-face meetings to review test questions. The column describes the use of a modem to conduct a conference over a single telephone line that shares voice and computer data simultaneously.

"Legal Beat" by Dale Atkinson

Atkinson describes the legal concept of collateral estoppel and its implications for administrative proceedings against licensees in regulated professions. He also discusses a case in United States District Court, Northern District, involving the applicability of Title VII of the Civil Rights Act to voluntary certification agencies, a first of its kind.

"The Expectations of Standard Setting Judges" by Gerard F. Dillon

Dillon discusses how the results of standard setting can sometimes be viewed as unacceptable by relevant stakeholders. He describes a technique he believes can be used to reduce this problem that can also be helpful in training judges participating in standard setting exercises.

"Item Harvesting: An Efficient Method for Generating Test Questions" by John Norcini, Paul
         Poniatowski, Susan Day, and Elizabeth Callahan

The authors describe a procedure, called "item harvesting," for producing test questions. The procedure solicits the active participation of large numbers of practitioners, is carried out by mail and does not require a large-scale, face-to-face test committee meeting to review all of the questions.

"Test Validity Evidence: What About Face Validity?" by Steven M. Downing

Downing discusses the concept of face validity and its misuse, particularly in performance testing. This article emphasizes that more evidence is needed to demonstrate the validity of an examination than just its apparent fidelity to the job.

CLEAR Exam Review, Winter 1996, Volume 7, No. 1

"Testing Across the Nation" by Barbara Showers

Showers indicates that the need for examination security seems to be greater than ever. Her column describes some of the sophisticated devices now available to potential cheaters as well as steps credentialing agencies can take to improve test security in the face of these threats.

"The Answer Key" by Norman Hertz

Hertz's column raises and answers four questions. The first responds to a question concerning how to evaluate the quality of a criterion-referenced passing score study. The second concerns the implications for testing when the results of an occupational analysis indicate that activities not mentioned in the regulations are being performed by practitioners. The third question deals with the issues involved when a board considers using a performance test as part of its licensing process. The last question deals with concerns regarding test standards and whether they are consistent from state to state.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer identify and summarize a number of papers, articles, and reports reflecting current thinking on performance assessment, situations encountered in adapting performance assessments to licensing/certification applications, and historical perspectives of performance testing.

"Software Review" by Lee Schroeder and Pansy Dubose Houghton

Schroeder and Houghton discuss how the use of computers in examination development processing, analysis of test results, and the storage of confidential information requires the design and implementation of tough security procedures in order to restrict access to appropriate personnel.

"The Death of PMPs and the Lesson for Performance Assessment" by Lynn C. Webb

Webb's column discusses the factors that caused the death of PMPs. She describes the psychometric difficulties as well as the financial and timing issues that resulted in the demise of this technique.

"Legal Perspectives of Examination Security" by Dale Atkinson

Atkinson's article deals with the legal aspects of test security. The author discusses the ways in which the security of examinations can be breached. He then describes the procedures and legal actions boards and examination contractors can take to protect themselves and the public as well as discouraging individuals and organizations from attempting to breach test security.

"Licensing and Certification Test Construction: A Balancing Act" by Wade Gibson and John Weiner

This article describes the procedures that can be used to ensure that alternate forms of licensing and certification examinations meet both content and psychometric standards. The approach used is based on classical test theory and can be used with computerized tests to enable users to administer a unique alternate test form to each candidate.

CLEAR Exam Review, Summer 1995, Volume 6, No. 2

"Testing Across the Nation" by Barbara Showers

Showers covers four themes: the Americans With Disabilities Act and it focus on describing the "essential functions" of jobs; the National Board of Examiners' decision to stop using the latent-image gathering format in the Patient Management section of its certifying exam; the security concerns regarding computer-based tests; and the State Post-secondary Review Entity, which may require regulators to provide information on passing rates for institutions receiving Title IV funds that have educational programs leading to licensure.

"The Answer Key" by Norman Hertz

Hertz's column raises and answers five questions. The first concerns how examinations are affected when interpreters are used to translate an examination into a language other than the one in which it was written. The second asks what can be done to ensure that more items survive an item writing workshop. The third discusses the options a board has other than accepting passing scores that have been established by a national association. The fourth asks how to evaluate the quality of test questions. The last one is concerned with the meaning of validity in a licensing examination.

"The Americans with Disabilities Act Versus Your Board's Duty to Investigate Applicants: The ADA is
      Winning!" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L. Bloom

This article discusses the Americans With Disabilities Act and licensing boards' duty to investigate applicants. It is clear that boards need to be very careful about the questions they ask. Useful advice is provided on the kinds of questions that are appropriate and can be defended.

"Abstracts and Updates" by Charles Kunce and Mary Sandifer

Kunce and Sandifer review: guidelines for computerized adaptive tests, an article describing procedures designed specifically for estimating the reading level of test questions, an article and paper describing the effects of changing answers on multiple-choice examinations, as well as two articles dealing with setting passing scores.

"Software Review" by Lee Schroeder and Pansy Dubose Houghton

Schroeder and Houghton's column notes that bringing subject matter experts together to write and review examinations is one of the major expenses associated with an ongoing testing program. Their column describes test questions by computer and by telephone.

"America's School-to-Work Initiative: What Role Should Credentialing Agencies Play? " by Meredith
        Mullins and Donald Ross Green

Mullins and Green discuss the current emphasis on improving the transition from school to work. They point out what they believe are some of the unique capabilities and experiences that credentialing agencies could contribute to the solution of this national problem. And they raise questions about the role of credentialing agencies in this area.

"Development of Performance Assessments for Use in Professional Certification and Licensing" by
        Janice Dowd Scheuneman

Scheuneman's article explores some of the difficulties that threaten the validity of performance measures. These include the lack of generalizability of scores and construct irrelevant variance present in scores.

"Psychometric Evaluation of Performance Assessments: Critical Issues and Innovative Strategies" by
        Richard M. Jaeger

Jaeger's article describes the development of performance assessments by the National Board for Professional Teaching Standards (NBPTS). The NBPTS is developing performance assessments in a number of content areas that will be used to certify accomplished teachers. This article discusses the approaches used to develop the certification for Early Adolescence Generalists. It describes a variety of critical psychometric issues that arose in the development of these measures as well as the strategies used to address them.

CLEAR Exam Review, Winter 1995, Volume 6, No. 1

"Testing Across the Nation" by Barbara Showers

Showers discusses the use of computerized testing in licensing and certification. Barbara describes the various computer-based testing options, administration of computerized tests, impacts of computerized test delivery on the credentialing program, justifying the change, justifying the increased cost, availability of providers, components of cost, and the planning necessary for implementing computer-based testing.

"The Answer Key" by Norman Hertz

Hertz's column raises and answers five questions. The first deals with how to maintain the same level of scientific rigor in constructing the examination plan as was used in conducting and analyzing the job analysis. The second asks about the guidelines that can be used to define the entry-level practice when conducting a passing score study. The third deals with the factors to consider when developing procedures for scoring performance, problem, essay, or oral examinations. The fourth involves the role of board members in the examination development and scoring process; the fifth considers the role of educators in the examination development process.

"Unconstitutional Advertising Restrictions" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L. Bloom

This article discusses the type of advertising restrictions that licensing agencies have adopted that have been upheld by the courts as well as the type of restrictions that may be unconstitutional.

"Abstracts and Updates" by Mark Raymond

Raymond's column discusses new publications in three major content categories. The first category is concerned with item writing. The articles in this section cover topics such as estimating the optimum number of options and the effects of altering the position of options in multiple-choice examinations as well as use of taxonomies in test construction. The second deals with procedures for use in estimating the reliability of performance tests. The third category includes articles on job analysis, testing policy, and computerized adaptive testing.

"Software Review" by Lee Schroeder and Pansy Dubose Houghton

Schroeder and Houghton's column describes two software products they believe can very useful to regulatory agencies. One is Microsoft Access version 2.0, which is a database management tool. The other is DesignExpert, which enables an agency to produce its own scannable documents.

"Accommodating Different Ethnic and Cultural Groups in Credentialing Examinations" by Meredith
        Mullins and Donald Ross Green

Mullins and Green discuss the importance of global awareness and sensitivity to cultural diversity in the design and administration of today's credentialing examinations.

"Evaluating Licensure and Certification Examination Programs" by Steven M. Downing and Thomas
        M. Haladyna

Downing and Haladyna describe how high-stakes licensure and certification programs can benefit from external review. They discuss the advantages of such reviews, the aspects of the credentialing program that should be reviewed, the standards that should be considered for evaluating the program, as well as criteria to use when selecting an outside evaluator.

"Expanding a Professional Assessment Program: Issues for Cross-Cultural Testing and Test
        Translation" by Lynn C. Webb

Webb's article notes that we have entered the age of the global community and that many certification and licensure programs are investigating the possibility of expansion. Lynn discusses issues and concerns related to cross-cultural testing and test translation. She emphasizes the need for planning, the use of appropriate testing standards, translation strategies, and special areas of inquiry.

CLEAR Exam Review, Summer 1994, Volume 5, No. 2

"Testing Across the Nation" by Barbara Showers

Showers discusses why she believes sole-source providers of credentialing examinations concerns about contract clauses, policy issues, and options for states. Barbara also discusses the progress being made on revising the Standards for Educational and Psychological Testing promulgated by the American Psychological Association, American Educational Research Association, and the National Council for Measurement in Education. Barbara also provides a list of organizations and publications that can be used as resources for issues related to performance-based assessment.

"The Answer Key" by Norman Hertz

Hertz's column raises and answers five questions. The first deals with whether licensing examination programs should provide diagnostic feedback to candidates who do not pass examinations. The second asks if a passing-score workshop is necessary to establish a criterion-referenced passing score for every examination--even if a substantial percentage of test questions are reused from administration to administration. The third concerns factors to consider when selecting subject-matter experts so as to ensure they represent practice. The fourth concerns issues related to the use of oral examinations; and the fifth deals with the appropriateness of subject-matter experts writing test questions in advance of item-development workshops.

"Licensure Denials: Know Your Authority" by Susan E. Dorn, Kim A. Zeitlin, and Margaret L. Bloom

This column discusses the importance of a board knowing its authority under its state's administrative procedures act when imposing disciplinary procedures.

"Abstracts and Updates" by Mark Raymond

Raymond's column discusses six articles concerned with the validation of licensing and certification examinations; six articles dealing with item-writing and test assembly; five articles concerned with alternative assessment; and three articles related to legal and administrative issues.

"Software Review" by Lee Schroeder and Pansy Dubose Houghton

Lee Schroeder's column applies his criteria for evaluating an item-banking system (published in a previous CER column) to two item-banking systems that were sent to him for review. One of Lee's associates, Pansy Houghton, reviews LXR-Test 5.0 and The Question Bank.

"Gender Bias in Licensure Testing: Is It a Problem" by Meredith Mullins and Donald Ross Green

Mullins and Green discuss the possibility of gender bias in licensure testing.

"Evaluating Test Fairness in Licensure Testing: The Sensitivity Review Process" by Stephen G. Sireci
        and Laura A. Mullane

Sireci and Mullane discuss the sensitivity review process as one aspect of evaluating test fairness in licensure testing. They describe the purpose of sensitivity review and the sensitivity review process, and also provide examples of problematic items as well as a sample rating sheet for documenting the results of a sensitivity review.

"Providing Interpreters for Deaf and Hard of Hearing Candidates for Credentialing Examinations" by
        Catherine Nelson

Nelson's article discusses issues related to providing interpreters for deaf and hard of hearing candidates for credentialing examinations. She outlines an interpreter's responsibilities, as well as some of the decisions a board needs to make when using interpreters. Catherine also identifies a source boards can use to identify qualified interpreters.

CLEAR Exam Review, Winter 1994, Volume 5, No. 1

"Testing Across the Nation" by Barbara