April 9, 2019

Line Dempsey: Welcome back to another one of our podcasts, Regulation Matters: a CLEAR conversation. Once again, I'm your host, Line Dempsey. For those that may not know me, I'm the senior investigator with the North Carolina Dental Board and I'm on the CLEAR board of directors, as well as the current chair of the NCIT Committee and vice chair for the annual conference program committee with CLEAR. As you may also know, the Council on Licensure, Enforcement and Regulation, or CLEAR, is an association of individuals, agencies and organizations that comprise the international community of professional and occupational regulation. Our podcast is a chance for you to hear the latest and greatest in our community. Today, I'm happy to be joined by Elizabeth Witt, chief consultant and psychometrician with Witt Measurement Consulting. She holds a PhD in Psychometrics and has worked in licensure and certification testing for approximately 20 years. We're glad to have you with us; welcome.

Elizabeth Witt: Thank you; I'm glad to be here.

Line: And again thanks for joining us. The topic we'd like to talk about today is making the most of expertise, guidelines for working with subject matter experts. In licensure and certification testing, subject matter experts, or SMEs, are crucial participants. The validity of an assessment program depends heavily on the involvement of the SMEs. I guess let's start with the question that is out there, how do you define a subject matter expert?

Elizabeth: Well, in the context of licensure and certification testing, a subject matter expert is a person with expertise in the specific field or profession who can also apply that expertise in helping us create examinations that are valid for the purpose of screening new applicants for licensure or certification. So knowledge of the content of the field is required obviously, but more than that, we want people to come in with knowledge about the job, what's done on the job, how has it operated. We want them to have knowledge of what's covered in education and training in preparation for a career in this profession. We want them to have knowledge of ethics and professionalism.

Typically, when we're looking for an SME we're looking for someone who already holds the credential we're testing for. They may be brand new to the field or they may be highly experienced, depending on the task we seek their help with. And it helps if they also know something about testing, about standard setting, if they have some experience in doing these things before. But it's not absolutely necessary because we're able to build that kind of knowledge into the activity that they're going to be participating in.

Line: Well, I guess kind of continuing on that, what kind of qualifications does an expert have to have in developing, administering, scoring and examination? We'll get to, I guess, their roles when you look at licensure and certification in just a moment. But I guess, do they need to have been practicing or licensed for a certain number of years? What kind of things kinda play into that?

Elizabeth: Well, to some extent it depends on the task you're asking for their help with. In general, obviously, you need somebody with expertise. They need familiarity with the knowledge base of the field; they need familiarity with the challenges of the profession; they need familiarity with factors that are risky for safe and effective practice.

You also wanna get people who have good people skills. They should be able to work well with others. You don't wanna recruit a prima donna; you wanna recruit people who can make a positive contribution without trying to control the process or the outcome of the task.

And we also want to consider a group of subject matter experts as a whole. We're looking to balance diversity and representativeness. So if you have a profession that has a lot of different sub-specialties - maybe people work in different practice settings or serve different client groups such as children versus elderly; or they come from different geographic regions; or just demographic characteristics like gender, age, years of experience - all of these are things that can affect people's opinions, and we want a balance of diverse opinions. So diversity and representativeness are actually more important than aiming for a specific number of people to serve on a committee of SMEs.

Now, if we could talk about some specific context - If you're looking at gathering SMEs to perform a job analysis or practice analysis, it's best if you have at least some people who have experience across a variety of different settings, so they get that broad perspective. Most of your SMEs on a job analysis committee will probably have many years of experience, but it's nice if you can to get at least one or two people who are new to the profession because these are people who will have familiarity with new techniques and new technologies that people who've been out in the field for a long time may not really be aware of. Diversity - obviously, it's important. We don't wanna overlook a sub-specialty, for example, or a geographic region where things are done differently. And on a job analysis committee, you could include a board member or two; you just don't want to make board or staff the majority. And I would say the same for educators. It might be good to have an academic or two on a job analysis committee, but you don't want them to override everyone's decision.

Another area in which we use SMEs is item writing. In this case, you want people to have a strong and broad knowledge of content, and you want them to be familiar with the resources or reference materials that are used in the profession. It helps a lot if they have strong writing and editing skills. It helps if they have strong analytical skills, but these aren't really always necessary because we've found that we can train most SMEs to write items, pretty good items actually. Often some people will come in with a knack for it; they're just born good at it. And occasionally you get someone who's just not going to get it, so you take what items they can write and you just don't invite them back next year. But in general, we found that SMEs are intelligent people, they're dedicated to their profession, and we can train them fairly easily to write some pretty good test items. When you're looking for item-writing SMEs, you also want people who can make a long-term commitment, people who are dedicated to their profession, who may be willing to train or mentor others down the road as they continue to increase their own skills.

Another setting in which we use people is standard setting. This means setting the cut score on the examination. In this case, you definitely want to make sure that your group is representative of the expertise of the profession. You want some people who have a very broad experience, and you want some people who are very specialized. At the same time, you wanna be careful not to let the sub-specialties override the general practice because you're gonna be setting a passing score on an examination and it's an examination that has to be taken by all candidates, no matter what area they're going into. People on a standard setting committee need to have strong knowledge of content - skills as well as knowledge. They need some familiarity with the requirements of the profession at entry level - what do people need to be able to do, what do they need to know when they first start practicing in a profession? They should have familiarity with the education and training that candidates go through. And it helps if they're aware of the typical behavior of new professionals. So it's nice if you can get people who work with someone who's actually new to the profession, and even if you can get one or two people on the committee who are new to the profession themselves.

I often ask people as standard setters how many of them have familiarity with someone who is minimally qualified, or borderline qualified? And the reason I ask that is because, if you can get a picture in your head of what a minimally competent person looks like, that's actually very helpful in determining what a hypothetical person would do, who's at that minimum level of competence, when they encounter a particular question or task. On a standard setting committee, you might want to include people who are educators, academics. You might want to include people who are supervisors. Again, though, you want the majority of your committee to be made up with people who are actually out there doing the job of that profession. The standard setting committee is one area where you do not want to include a board member or a staff member of the organization as a voting member of the standard setting committee. These people can be present in order to provide information or clarification, but they should not influence the outcome of the standard setting. That might be viewed as a conflict of interest. In most cases, the board is going to consider the standard setting committee's recommendation and they can confirm or change it afterwards.

People serving on standard setting committees also need to have an understanding of the credentialing program and an understanding of the purpose of the exam. They should understand the consequences of passing or failing and the need for fairness. They should also understand what makes a task or a question difficult. They should know what standard setting is and how it works. And, fortunately, all of this information is material that we can cover in the training on the first day of a standard setting meeting.

Line: Gotcha, now you mentioned earlier, I think, something that was interesting to me because we often have to get experts to testify in cases and we try to recruit them (and I'll get into recruiting in just a second), but I thought it was interesting to point out that having maybe newer licenses involved in that process for a subject matter expert is important since they're getting a lot of the new fresh things. And that's something that typically, we've always gone with somebody that has been in the field or in the practice for a long time. They've got this wealth of knowledge, but I think it's a really credible point to consider someone that is new in the field that's coming off the latest and greatest as far as techniques being taught in school. Or sometimes even I've just experienced they have a little bit more interest in that particular topic at the start of their career, which bridges to the next question I've got, which is how do you recruit these type of subject matter experts? What do you do to motivate them? Do you pay them? How do you get them out of their little niche into an area where they're helping as a subject matter expert?

Elizabeth: Probably the best way to recruit new SMEs is by referrals, especially if you can get referrals from people who have already served as SMEs and they understand the task. So you're looking for people who are interested in the task, dedicated to their profession, and very capable. And fortunately, most such people respond very well to intrinsic rewards. Sometimes organizations will pay their SMES, but whether or not that's a good idea may vary by the profession. And before you collect a group of SMEs, you might actually wanna survey your population and see how they feel about that. I know in some professions, the SMEs were insulted by an offer to pay them because, for a lot of people, just the recognition and prestige is a great reward in itself. Just being invited to participate is an honor. Now in other fields, you may want to offer at least a modest honorarium to cover their time. You don't have to pay them necessarily as much as they make when they're working, but to give them something that covers the time that they spend coming to meet with you, or even working on things at home. This is especially true, I think, in professions where they have to take unpaid time off work in order to come and participate, or if they don't really make a particularly high salary to begin with. It helps a lot if you can compensate them for their time.

If your organization can't really afford to compensate them financially, sometimes you can reward them in other ways. A lot of organizations offer continuing education credits or perhaps they would exempt their SMEs from having to take a recert exam on the next round or offer reduced recertification fees. At the very least, you want to cover your SMEs' expenses, especially their travel expenses. If they come together with you for a meeting, you'll want to provide meals during the time that they're there. And it's very typical that if a meeting lasts for two or three days that you will take them out to at least one nice dinner during that time. Any perks that you can give them that helps them see that, yes, you really appreciate the contribution that they're making to your organization.

Line: Well, once you've got them recruited and working for you, how can you feel confident, or be assured that they are engaging in the tasks that you need them to do? How can you ensure that it's also a valuable experience for them?

Elizabeth: I am actually a pretty strong component of face-to-face meetings, especially for meetings that last longer than an hour or two. Working remotely, sometimes you can save money, but sometimes it's not in the best way. People can run into problems with distraction and inattention, and you just simply lose that group dynamic that is quite important in many situations. If you've ever sat through a meeting on a conference call that lasts more than an hour, did you maybe find yourself checking your email or otherwise multi-tasking?

Line: Of course.

Elizabeth: I just don't think you can ask your SMEs to sit there for a two-day meeting on the phone or on a computer screen. It's just impossible to do that without getting distracted and doing other things. Now, there's nothing wrong with having conference calls that last an hour where everybody is just kind of checking their email, but when it comes to the kind of work that you are asking SMEs to do, you want them to be on task for the entire time. You want them to be paying attention to the entire meeting and to the comments that are made by the other SMEs.

So I think a face-to-face meeting is particularly important when you're working on something like job analysis or standard setting. And another advantage of it is that the SMEs create better rapport with one another and they sometimes form lasting professional relationships with the other SMEs as well as with your organization.

Another thing you can do to keep people engaged is make sure you communicate with them regularly. Make it a priority to build good relationships with your SMEs, whether you have them just doing a one-time task or whether it's an ongoing relationship, such as you might have with item writers. You wanna do everything you can to make the experience of participating meaningful and interesting. And it is, in most cases, a professional growth opportunity, and most SMEs are going to appreciate that. If you can provide meals or other social activities, you're providing opportunities for them to get to know one another off task.

Make sure you always sincerely express your appreciation for the contributions, both in person and in writing. Let them know that their work benefits your organization and let them know how it benefits your organization.

Line: Well, I guess as one kind of final question I guess, what challenges might be involved in managing your subject matter experts?

Elizabeth: Ah, Murphy's Law. If anything can go wrong, it will, and it will most likely be something you did not expect. So you're bound to run into things like technology glitches, power outages, travel disruptions. And travel disruptions sometimes keep one SME from getting there, and that may not be important depending on how key they are to your process. Sometimes the travel problem can become a problem for everyone - so if there's a blizzard and no planes can land in the town where you're holding the meeting. So it's a good idea to have some sort of back-up plan. You can also have things happen like one or more SMEs can't make the meeting due to a personal situation, an illness or a family emergency. Once in a while, you get an SME who doesn't seem to quite understand the task. And if you're working on something like a job analysis meeting or a standard setting, you can actually consider omitting that individual's influence on the final outcome after the meeting. Or an item writer that just isn't working out; you could simply not ask that person to continue next year. Once in a while on rare occasions, an SME might be disruptive in one way or another. It might even be something as simple as falling ill during the meeting, and the meeting itself is disrupted.

So I guess my motto is, expect the unexpected; always try to have a backup plan; and, above all, keep a sense of humor.

Line: Very good. Hope for the best, but prepare for the worst.

Elizabeth: Yes, exactly.

Line: I appreciate you taking your time out today to be a part of our podcast. It's always exciting and wonderful to have the opportunity to share and learn from each other. I know I've gotten some things from this conversation today and so I certainly do appreciate it. Elizabeth: Sure. Line: And thank you to our listeners for being involved again with us. We'll be back with another episode of Regulation Matters: a CLEAR conversation very soon. Please subscribe to our podcast, which as you may know is available on a lot of different media: Podbean, iTunes, Apple Podcast, Google Podcast, Google Play, Stitcher, Spotify or TuneIn. And if you've enjoyed this podcast episode, please leave a rating or comment in the app. Your reviews help us improve our ranking, and it helps make it easier for other people and new listeners to find us. Feel free to visit us at our website at www.clearhq.org for additional resources and a calendar of upcoming training programs and events. Finally, I want to thank our CLEAR staff, specifically Stephanie Thompson. She is our content coordinator and editor for this program. I'm Line Dempsey, and I hope to be speaking to you again soon.

The audio version of this podcast episode is available at https://podcast.clearhq.org/e/working_with_SMEs/

Copyright © 2021 CLEAR
Please read the following disclaimer.  |  
Website by Nicasio LLC and CLEAR
108 Wind Haven Dr., Ste. A
Nicholasville, KY 40356
(859) 269-1289
Fax: (859) 231-1943
Powered by Wild Apricot Membership Software