EVALUATING A NATIONAL EXAMINATION PROGRAM

The National Association of State Boards of Accountancy (NASBA) is the association of boards of accountancy in all 54 U.S. jurisdictions (the 50 states, DC, Guam, Puerto Rico, and the Virgin Islands). Under the accountancy statutes of the various jurisdictions, each board of accountancy is responsible for the administration of an appropriate examination for applicants for CPA licensure. To test the technical competence of CPA candidates every board of accountancy uses the Uniform CPA Examination and the Advisory Grading Service provided by the American Institute of Certified Public Accountants (AICPA).

The relationship among the state boards of accountancy, NASBA and the AICPA is unique. In most licensed professions, the licensing examination is provided by the organization which represents the state boards, such as NASBA in the accounting profession. However, as in other professions, the boards of accountancy are responsible for all decisions relative to the licensing examination.

Although every state board of accountancy uses the CPA examination, each board is responsible for assessing the quality and content of the examination and for assuring that the grading and administration of the examination are satisfactory. This responsibility has been the focus of much attention since the advent of legislative oversight which called for the review, evaluation, and justification of all licensing requirements, and which also required state regulatory boards to demonstrate the appropriateness of their licensing examinations. However, few boards have the resources to evaluate the quality and content of the examinations or to review their preparation, scoring, and administration. Such evaluation and review are highly technical and time-consuming. Clearly they can be performed more effectively and economically by a single entity acting on behalf of all the boards than by each of the boards acting independently. Because NASBA is answerable to all boards which are its constituent members, it became the logical entity to provide this service. In the late 70s NASBA and AICPA leadership recognized that NASBA had a legitimate interest in the CPA examination and should conduct reviews on behalf of boards rather than have each board conduct its own review. In 1975, a CPA Examination Review Team was conceived as a joint venture between the AICPA and NASBA and reported to the AICPA Board of Examiners (BOE). In 1977, that review team was recreated into a independent body of NASBA which reports directly to boards of accountancy and was renamed the CPA Examination Review Board. The AICPA proposed that NASBA’s ordinary operating expenses, including provision for its own office facilities at a distinct location and for the cost of the annual review of the CPA examination, be financed by an allocation of a portion of the CPA examination fees paid by state boards to the AICPA. NASBA’s Board of Directors generally agreed with this proposal and, at a meeting in July 1978, the AICPA’s Board authorized its implementation. An agreement was subsequently signed by the two organizations.

It was agreed that the CPA Examination Review Board would conduct a comprehensive annual review of the CPA examination process to assist boards of accountancy in discharging their responsibility for the examination. Although a unique relationship exists in accounting, this model for evaluating an examination program can be applied to any professional examination by an independent board or committee, composed of knowledgeable, dedicated representatives of the profession.

The Review Board’s charge is as follows:

"The CPA Examination Review Board shall review, evaluate, and report on the appropriateness of the policies and procedures utilized in the preparation, grading, and administration of the Uniform CPA Examination and other examinations in general use by boards of accountancy for the licensing of certified public accountants. In carrying out its responsibilities, the CPA Examination Review Board shall examine such records, and make such observation, inspections, and inquiries as it deems necessary. The CPA Examination Review Board shall report annually to the boards of accountancy, the constituent members of the National Association of State Boards of Accountancy."

The existence of the Examination Review Board has helped insure the integrity and creditability of the CPA examination process and has assisted boards of accountancy in fulfilling their statutory obligations. The work of the Review Board has been cited in sunset review reports, at legislative hearings, and at various accountancy board hearings when the validity and reliability of the Uniform CPA Examination have been addressed. The Review Board has had a constructive influence on many projects which have led to the continuing improvement and defensibility of the examination.

For instance, until 1996, the CPA examination was disclosed. With the implementation of a nondisclosed examination in 1996, it is now possible to pretest questions and develop an item bank of questions from previously administered examinations. It was the Examination Review Board that urged in several of its annual reports and in a white paper, that the examination become nondisclosed. The Review Board served as a catalyst for many of the other major changes in the examination program such as: the implementation of a new passing standard; institution of item writer workshops; development of test banks of pretested questions; computerization study; and the establishment of the Examination Committee-- a vehicle for the implementation of the recommendations made by the Examination Review Board in its annual management letter.

In accordance with its charge, the CPA Examination Review Board is appointed by NASBA’s Board of Directors. Nine volunteers made up of state board members and representatives from academia devote about 90 days each during the year visiting board offices and examination sites, and reviewing the procedures of the AICPA Examinations Team. It divides its work into five principle areas: BOE policies and procedures; technical assessment; preparation; administration; and grading. The Review Board engages an independent testing consultant, a slot currently held by Dr. Steven Downing, who is the senior psychometrician at the American Board of Internal Medicine.

The Review Board begins its work program by reviewing the policies and procedures of the AICPA’s Board of Examiners. The Examinations Team Director is asked to complete a comprehensive questionnaire which provides the Examination Review Board with updated information about the Examinations Team policies and procedures. Based on a careful review of this questionnaire, the Review Board makes appropriate changes to its work programs. For instance, when the Content Oversight Task Force was established, a Review Board member was assigned its liaison. If new procedures are being implemented, the Review Board may create a work program for its review.

The technical assessment is completed by the consultant and is based on guidelines and standards derived from Standards for Educational and Psychological Tests and Generally Accepted Psychometric Principles. In addition, members compare the examination items to the content specification outline. Evidence of correlations among the scores on the four sections of the examinations are also evaluated.

To assess administration and security by boards of accountancy, the Review Board visits board offices and examination sites on a regular basis to monitor compliance with the procedures set forth in NASBA’s Handbook for CPA Examination Administration. These site visits also ensure that the examination is being administered in a uniform manner, wherever it is administered, based on an agreed upon set of procedures.

Since the inception of the program, members of the Review Board have conducted numerous reviews at many examination sites and at board offices. In 1997, members of the Review Board visited 31 examination sites in 17 jurisdictions to observe the conduct of the examination. Prior to visits, each jurisdiction is asked to complete two self-review checklists relating to the administrative procedures used on at the board office prior to the examination, and the procedures used on site to conduct the examination.

Following these visits, a report is issued to each jurisdiction commenting on the Examination Review Board member’s observations. There are four levels of reports issued, two of which require a board response to the Review Board. Follow up site visits are occasionally prescribed, along with requests for self-reviews. Almost all of the 54 boards are visited once every three years. A remote location, such as Guam, is visited less frequently.

In connection with administration security at the AICPA, the Review Board assesses the procedures relating to the security clearance of personnel, handling of examination drafts, printing, packaging and distributing examination booklets, designing and packaging examination answer materials, controlling candidate answer papers, and recording and reporting examination grades. It also observes the procedures and system of security at the examination preparation and grading facility, as well as at the facilities where the examination is printed.

The Review Board also carefully monitors the examination grading process. Review Board members meet with the AICPA examinations staff and observe the work of grading section heads, reviewers, and production graders. A Review Board member attends the Standard Setting Subcommittee meetings, held two weeks after each examination. At this meeting, the BOE reviews the sample grading statistics so that production grading may be done, reviews problem questions, and approves the grading bases. Based on the outcome of this meeting, along with a technical review of the preliminary data, the Review Board issues a supplementary report on grading which provides boards with limited assurance that they may rely on the grading of the examination. The timing of this report is such that, although the review is limited in scope, (because it is based on preliminary statistics), the state boards receive the report prior to their approval of grades. We call this report the "comfort letter," as it is designed to provide some comfort to the state boards on the grading process. Introduced in 1996, these reports, issued within 60 days of each examination, provide a service to boards that need to approve grades prior to their distribution.

The Review Board takes all of this information, analyzes it and synthesizes it into an annual report which opines on whether boards may rely on the examinations in carrying out their licensing responsibilities.

Keeping with the traditional audit approach, the Review Board also issues a management letter which identifies issues which require attention. The Examinations Committee (acting as "management" on behalf of state boards) reviews the comments and recommendations of the Review Board, (the auditor), and seeks resolutions of the issues. Here are some illustrations.

As noted earlier, it was the Examination Review Board that recommended the adoption of a nondisclosed examination which was finally implemented in 1996. For several years, the Examination Review Board had noted in its management letters that the inclusion of essay questions does not contribute to the validity of the examination; they also urged that all objective questions be pretested, not only the multiple-choice variety. With the assistance of an Examinations Committee, such observations and recommendations are addressed and implemented.

In visits to examination sites, Examination Review Board members identify problems, as well as sound procedures, that might be shared with all boards of accountancy. For example, with emphasis on examination security, the Review Board has praised the use of shipments of examination materials by armored car service but has also observed a flaw in the system used by some boards. Although some examination materials are delivered by armored car service to board offices, some board representatives use personal autos to transport them to the examination sites. With the identification of such security breaches, the board in question would consider alternative procedures when it receives the Examination Review Boards evaluation of the visit. Without identifying the particular board, all boards become aware of the flawed procedure when it is included in the Examination Review Board’s management letters.

As you can see, the work of the Review Board has provided a real service to the state boards of accountancy and the examination provider, and has made a number of important changes to the examination program.

COPYRIGHT 2000. Rights to copy and distribute this publication are hereby granted to members of the Council on Licensure, Enforcement and Regulation (CLEAR), providing credit is given to CLEAR and copies are not distributed for profit.