Job Analysis: A Guide for Regulatory Boards
by Roberta Chinn and Norman Hertz


TABLE OF CONTENTS


WHY CONDUCT A JOB ANALYSIS?

  • Purpose of a Job Analysis
  • Content Validation Strategy
  • Professional and Legal Standards

ASSUMPTIONS

  • Purpose of a Licensing Examination
  • Public Protection
  • Entry-Level Perspective
  • Generalist Versus Specialist Practice
  • Level of Specificity

COMMON METHODOLOGIES

  • Role Delineation
  • Multiple Methods Approach

SOCIAL AND POLITICAL CONSIDERATIONS

  • Economic Cost
  • Resistance to Change
  • Existing Statutes and Regulations
  • Educational Curriculum
  • Competing or Overlapping Scopes of Practice

COMMON QUESTIONS AND ANSWERS

REFERENCES

OTHER RESOURCES

 

Why conduct a job analysis?

Purpose of a Job Analysis
The primary purpose for conducting a job analysis is to define practice of a profession in terms of the actual tasks that new licensees must be able to perform safely and competently at the time of licensure.

The definition, or, description, of practice is typically used to form the foundation of a licensing examination that is fair, job-related, and legally defensible. The definition of practice can also provide a solid foundation for developing legislation and policies that affect regulation and educational standards. A job analysis should be routinely reviewed every five years to maintain an accurate description of practice.

Content Validation Strategy
The approach of choice for validating a licensing examination is called the content validation strategy. The content validation strategy establishes job-relatedness of the examination and thereby provides empirical linkage of the occupational analysis to the examination content.

Establishing the validity of a licensing examination is a far more complicated matter than establishing the validity of any other kind of examination. Every profession has statutes and regulations governing the activities of licensees. There are aspects of practice too numerous to be comprehensively specified in statute or regulation. Thus, once the license is granted, a regulatory agency has very little control over a licensee unless blatant violations have occurred. Therefore, the job analysis provides concrete information about the profession that can only be otherwise assumed.

Professional and Legal Standards
The Standards for Educational and Psychological Testing (1985) and the Principles for the Validation and Use of Personnel Selection Procedures (1987) emphasize that job analysis is the primary basis for determining the content and knowledge crucial to protecting the public (Knapp and Knapp, 1995). Both the Standards and Principles are widely used as professional standards to evaluate the validity of examination programs.

Additionally, a number of statutes and guidelines, as well as case law, impact licensure testing. The statutes and guidelines include the federal Uniform Guidelines for Employee Selection Procedures (1978), the Civil Rights Act of 1991, and the Americans with Disabilities Act of 1990.

ASSUMPTIONS

Purpose of a Licensing Examination
A board must understand the purpose of licensing examinations in order to appreciate the scope and content of a job analysis. The sole purpose of a licensing examination is to identify persons who possess the minimum knowledge and experience necessary to perform tasks on the job safely.

By contrast, the purpose of academic examinations is to assess how well a person can define and comprehend terms and concepts; and the purpose of employment selection examinations is to rank order individuals who possess the qualifications for a job.

Public Protection
Licensure-related job analyses differ from traditional job analyses in a number of ways. The most important aspect of licensure-related job analyses is the focus on the critical competencies required to protect the public rather than on responsibilities and knowledge necessary for successful job performance (Kane, 1982). Activities related to professional development, supervision, in-service training, or business practice are usually left out of the job analysis because they are related to successful job performance rather than public protection.

Entry-Level Perspective
The job analysis should always include a number of newly licensed, entry-level practitioners to provide assistance during the establishment of content and criteria to define the profession. The inclusion of entry-level practitioners will ensure that the competencies addressed in the job analysis reflect professional situations commonly encountered by entry-level practitioners, and that examinations based on the job analysis reflect minimum competence.

Generalist Versus Specialist Practice
Many boards are concerned about how practice specialties are addressed in a job analysis. One strategy is to represent competencies for undifferentiated, generalist practitioners, although such persons are largely hypothetical (LaDuca, Downing, & Henzel, 1995). In addition, there should be a balance between frequently occurring, low-impact situations and infrequently occurring high-impact situations so that the job analysis is a comprehensive evaluation of practice. The job analyst must work very closely with subject matter experts to achieve the balance and identify differences among practitioners.

Level of Specificity
The results of a job analysis should provide sufficient detail so that a board can update examinations, policies, regulations, and curriculum. The key to obtaining useful job analysis results is to select a job analyst who is familiar with different job analysis methods that ultimately affect the level of specificity in the tasks and knowledge best suited for the board’s purposes.

Minor revisions of policy and regulations can be accomplished by evaluating the content of broadly stated, general competencies and duties. On the other hand, major changes in examination content, regulations, or curriculum can only be accomplished by evaluating the content of well-constructed tasks and knowledge that reflect precise technical concepts and terms.

COMMON METHODOLOGIES

Two methodologies---role delineation and multiple methods approach---are the most common methodologies to conduct job analyses for licensure purposes.

Role Delineation
The purpose of role delineation is to identify broad subject matter areas that include characteristics of tasks and their associated knowledge, skills, and abilities. Individuals who perform the job are asked to report their activities in different types of professional situations that define a domain of the profession. The information is synthesized into a survey questionnaire that is distributed to a large sample of practitioners. The data are analyzed to identify the core tasks and knowledge areas that are critical to competent performance. An example of role is "Direct Service" and a specific type of delineation of that role is "Plan, design, and implement intervention programs."

Multiple Methods Approach
The multiple methods approach adopted by Prien and his colleagues (Prien and Ronan, 1971) uses many of the elements of other job analysis methods in a systematic and precise way. The purpose of the job analysis is to differentiate individuals who perform tasks in a variety of job functions. Subject matter experts provide job information through individual interviews and focus groups. Job information includes detailed descriptions of job tasks and knowledge skills, and abilities necessary to perform the tasks.

The information is synthesized into a survey questionnaire that is sent to a large sample of individuals. In a typical survey questionnaire, individuals are asked to rate the relative importance of job tasks and their associated knowledge, skills, and abilities. The advantage of the multiple methods approach is that the job content provides sufficient detail to construct test questions.

An example of a job task for a land surveyor is "Perform topographical survey to produce a graphical terrain representation." An example of a knowledge statement is "Knowledge of elements of topographical maps to determine physical features."

SOCIAL AND POLITICAL CONSIDERATIONS

Social and political considerations can greatly impact the outcome of a job analysis and interfere with a board’s commitment to the job analysis process. The most important thing for a board to remember is that a job analysis is an empirical study of practice whose sole intent is to define a profession in terms of actual tasks that can be performed at the time of licensure. An empirical study should not be based upon the results of a survey sent to a professional organization’s membership, the recommendations of one or two advisory committees, or the results of previous job analyses.

Economic Cost
Most boards underestimate the cost of a job analysis because they do not fully understand its intent, purpose, and effect on licensed practice. Is a board willing to appropriate $50,000-$150,000 to carry out the project? Is a board prepared to devote a significant amount of time from its own staff to implement the project?

The cost is somewhat dependent upon the complexity of the profession and the number of practice issues, such as subspecialty practices, that can affect the number of personnel and the number of hours required to complete the job analysis. Generally speaking, job analyses of professions that require advanced degrees and prelicensure internships will cost more than professions that require vocational training.

If a board wants to conduct a job analysis, there are a few considerations that relate to cost. How many interviews will be conducted? How many focus groups are needed? How many survey questionnaires will be distributed?

Resistance to Change
There are also considerations that relate to how the concept of a job analysis will be received by board members and licensees. How will the results of a job analysis be used by the board? Does the board really want a job analysis or does the board want to confirm existing statutes and regulations? Is the board prepared to face the issues that the job analysis raises? Does the job analysis indicate major curriculum changes by educational institutions? If there are curriculum changes, how soon could the changes be implemented? How is the board perceived by its licensees? The best-planned efforts are doomed to fail if licensees do not want to participate in interviews or focus groups, or complete the survey questionnaire.

Existing Statutes and Regulations
Some professions have highly restrictive or highly specific state statutes and regulations that can impede how quickly the results of a job analysis can be implemented. And, many state governments have a regulatory process that can take two to three years to change the content of statutes and regulations.

Educational Curriculum
Because a particular subject matter is taught in educational settings, it does not follow that it should be included in a job analysis in a licensing examination. A job analysis of licensed practice should be more than a master curriculum outlined in an educational catalogue. Persons seeking a license must apply their training and education to actual tasks of the job and be able to perform those tasks in a manner that protects the public health, safety, and welfare.

There may also be differences in the scope and quality of different educational curriculum. Some programs are well rounded but have an outdated core of classes. Other programs overemphasize specific aspects of the profession to the exclusion of others. Still other programs have a satisfactory core of classes but have loosely constructed criteria for training and practical experience.

Competing or Overlapping Scopes of Practice
The difficulty of conducting a job analysis is compounded when related professions have scopes of practice that overlap within a profession. The job analyst conducting the job analysis must define the characteristics of the profession and determine how it differs from related professions. The job analyst will then have to establish common terminology, concepts, and contexts for that profession.

For example, licensed clinical social workers and marriage, family, and child counselors conduct psychotherapy. But the social worker conducts psychotherapy in the context of "person-in-situation" and the marriage, family, and child counselor conducts psychotherapy in the context of "relationships." During interviews and focus groups, the analyst should be consistent in the interpretation of "therapy" in the context of a specific profession so that the terms and concepts in the survey questionnaire are not misinterpreted.

COMMON QUESTIONS AND ANSWERS

Question

What topics should be covered in a job analysis?

Answer

A job analysis of a licensed profession should include only those subject matter areas that pertain to the protection of the public health, safety, and welfare rather than those that ensure an individual’s business success. What is important in a job analysis is what a minimally competent individual is observed doing rather than general business and recordkeeping skills or general communication skills. Thus, all of the activities included in the job analysis should be observable and have an impact on public health, safety and welfare.

Question

Our national organization has already performed a task analysis of our profession. Can’t we save some time and use this as the basis of our board’s job analysis?

Answer

A task analysis performed by professional organizations is often limited in scope and includes broad job duties that are either not observable or not testable. The scope may cover a range of duties that can exceed minimal competence or entry-level practice. These analyses usually do not include a broad range of licensed persons. Thus, the results of these analyses do not provide sufficient detail which can be used as the basis of test questions.

Question

Are job analyses conducted differently for occupational licensure than those for other purposes?

Answer

Yes. Job analyses, or occupational analyses, are designed to capture the tasks performed and competencies required at the time a person enters into the profession. This "entry-level" perspective affects the decision of whom to interview in terms of experience---for licensure a goodly proportion of the interviewees should have been licensed five years or less. The results from the job analysis are not intended to be used to identify complexities that would predict success in practice. The competencies that should be identified are those that relate to the protection of the public health, safety, and welfare.

Test providers are oftentimes accustomed to using job analysis results to develop tests that are predictive of success on the job. This approach is inappropriate for occupational licensing.

Question

We have been told that our occupational analysis is obsolete and that we should have a new one performed for our examination program to be considered content-valid. What kind of standards should we apply to evaluate the quality of an occupational analysis?

Answer

The basis for the construction of a content-valid examination is the job analysis. A job analysis is a comprehensive survey of various tasks and knowledge required to perform a job.

First, the list of tasks and knowledge that are developed from on-site interviews and follow-up workshops should be comprehensive. The development of the list should continue until no new or additional information can be obtained. Second, the level of specificity should be consistent for all tasks and knowledge. Furthermore, the statements should provide sufficient detail so that they are useful during the construction of test questions.

Tasks and knowledge that are broad and stated in a few words are not adequate for constructing new test questions because the subject matter experts must depend upon their personal experience to determine the context of the task or knowledge.

  1. There are a number of criteria in the professional literature and in case law that job analyses should adhere to for results suitable to produce examinations that are content-valid:
  2. The knowledge test must be important and not peripherally related to effective performance of job tasks;
  3. Subject matter areas of practice should be accurately weighted to reflect the relative importance of the attributes that they purport to test;
  4. The level of difficulty should match minimal competence; and,
  5. Job analysis interviews should cover the full spectrum of tasks performed by licensees.

Question

What is the role of educators in the job analysis process?

Answer

The role of educators should be minimized. Oftentimes, educators promote inclusion of specific subject matter that may not pertain to a job analysis that will be used for licensing purposes.

Question

What is the role of board members and professional associations?

Answer

A job analysis is not performed as a reaction to individual opinions, sentiments, or the whims of special interest groups. Therefore, the direct role of board members and professional associations should be minimized. A job analysis is an objective study of practice that should be free of influences from individuals or special interest groups. If a job analysis is done properly, the concerns of these groups should be addressed.

Question

Who should be interviewed for the job analysis? How many should be interviewed to make sure that all the information is covered?

Answer

Licensees from a range of practice settings, geographic location, and levels of experience can be interviewed. The purpose of the interviews is to provide the basis of the list of tasks and knowledge to be included on the survey questionnaire. Interviewees should be assured that the depth and breadth of the information obtained from the interviews is not the sole means of obtaining data about a profession and that panels of subject matter experts who represent the specialty mix of the profession will conduct an extensive review of the tasks and knowledge.

The number of subject matter experts to be interviewed depends upon the complexity of the profession and the issues involved in practice. For technical level professions, 10-15 interviews is usually sufficient. For those professions that require advanced degrees and supervised training programs, 20-30 interviews may be needed. The number of subspecialties or work settings may necessitate additional interviews.

Question

What should we consider in the design of the rating scales for the questionnaire?

Answer

A board should assist in the design of rating scales that answer questions of interest. Typically, job analyses ask respondents to rate how frequently they perform a task in current practice or how important a task or knowledge is to current practice. However, other rating scales can be used. For example, a board can determine the difficulty of acquiring a task or knowledge. A board can also determine if proficiency in a task was obtained before or after licensure, or how much supervision may be needed to perform a task safely.

Question

We think that too many rating scales may scare people from answering the questionnaire. Can’t we eliminate some of them?

Answer

While the length of the questionnaire may be of some concern, a board should be more concerned that all aspects of practice are included in the survey questionnaire. The outcome of the survey questionnaire is very difficult to interpret when valuable information about infrequently performed but essential tasks is omitted.

Question

Who should receive the questionnaire? We could ensure a high response rate by sending the questionnaire to people we know would fill it out.

Answer

When the results appear to be tied to specific individuals or groups of individuals, the results may be biased in favor of those individuals or groups or their special interests.

The strength of a psychometrically sound survey is the sampling strategy used to distribute the questionnaire. The questionnaire should be sent statewide to all counties. Statistical tables can be useful to determine the number of individuals to be included in the sample. Every effort should be made to send the questionnaire to the geographic, ethnic, gender, experience, and practice specialty mix of the profession.

Because the results of a job analysis focus on entry-level practice, half of the survey questionnaires should be sent to individuals who have been in practice for five years or less. The other half of the questionnaires can be sent to more experienced individuals, e.g., 6-20 years.

Question

We have completed an occupational analysis and used the results to develop a test plan for developing the examination. Doesn’t our examination meet validity requirements?

Answer

Not necessarily. Boards must do more than conduct a job analysis and develop a test plan. It is important to understand that an examination in and of itself does not possess validity. Validity of an examination is inferred if the examination tests job-related competencies established by the results of the job analysis.

Question

The scope of practice is specified in detail in our regulations. However, the results from an occupational analysis identified activities that are not mentioned in the regulations but are being performed by practitioners. What are the implications of the findings for examination development?

Answer

When the scope of practice is written in great detail in the regulations, it becomes very problematic to incorporate changes in actual practice in examinations. It may be that some recent aspects of practice cannot be tested until the regulations are changed. Moreover, practitioners may be performing services outside their accepted scope of practice. A long-term solution to the problem may be to change the regulations so they are stated in a more general way to make provisions for the future.

Question

How are test specifications established for occupational licensing examinations?

Answer

The results from the job analysis will identify the importance of the tasks and competencies for practice. Therefore, the most important tasks and competencies should serve as the foundation for the examinations. Technical standards indicate that it is appropriate to use only the tasks and competencies that are most important for practice when developing examinations. Occupational licensing examinations, as all examinations, can measure a limited set of tasks and competencies in the time allowed for the examination. Therefore, only the most important ones should be measured.

The validity of occupational licensing examinations are established by linking the questions directly to the tasks or competencies that have been found to be important to practice.

REFERENCES

American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (1985). Standards for Educational and Psychological Testing. Washington, D. C.: American Psychological Association.

Kane, M. T. (1982). The validity of licensure examinations. American Psychologist, 37(8), 911-918.

Knapp, J. E., & Knapp, L. G. (1995). Practice analysis: building the foundation for validity. In: J. C. Impara, (Ed.), Buros-Nebraska series on measurement and testing. Lincoln, NE: Buros Institute of Mental Measurements, 93-116.

LaDuca, A., Downing, S. M., Henzel, T. R. (1995). Systematic item writing and test construction. In: J. C. Impara, (Ed.), Buros-Nebraska series on measurement and testing. Lincoln, NE: Buros Institute of Mental Measurements, 117-148.

Prien, E. P., and Ronan, W. W. (1971). Job analysis: A review of research and findings. Personnel Psychology, 24(3), 371-396.

Society for Industrial and Organizational Psychology. (1987). Principles for the validation and use of personnel selection procedures (3rd Ed.). College Park, MD: Society for Industrial and Organizational Psychology.

Uniform Guidelines for Employee Selection Procedures. (1978). Federal Register, 43, 38290-38315.

OTHER RESOURCES

Browning, A. H., Bugbee, Jr., A. C., & Mullins, M. A., (Eds.). (1996). Certification: A NOCA handbook. Washington, DC: National Organization for Competency Assurance (NOCA).

Schmitt, K. & Shimberg, B. (1996). Demystifying occupational and professional regulation: Answers to questions you may have been afraid to ask. Lexington, KY: The Council on Licensure, Enforcement, and Regulation.



COPYRIGHT 2000. Rights to copy and distribute this publication are hereby granted to members of the Council on Licensure, Enforcement and Regulation (CLEAR), providing credit is given to CLEAR and copies are not distributed for profit.