I've done a lot of compliance courses. Those are often made to provide proof that someone has received info about a specific policy, procedure, etc.
Some clients think it's enough to restrict navigation, that is, to simply force users to step through all the slides. The successful completion status in an LMS can be considered proof that they received the info.
More often, clients also want a quiz, since that (allegedly) shows that the user learned something. These typically use standard question types like multiple choice. (Of course, writing good questions and answers is difficult. I think we've all seen "tests" that most people could pass without ever seeing the content.)
Evaluating training effectiveness is a whole other topic. For example, how would you define "effective" if the main goal is just to present info for compliance purposes? Plus, training is only one of many factors that impact whether someone actually follows a procedure. So people passing a quiz in an eLearning course might prove compliance with regulations re: providing info, but it can't prove the effectiveness of that info.
Thanks Judy, you raise really good points which have sparked additional questions for me to take back. For example, what is important to the client - are they looking to check a box that the training launched and that people completed?
Regarding the training effectiveness, with your experience developing compliance courses, have you had to evaluate and what do you think works, or what would you recommend.
I haven't been involved in training-effectiveness evaluations, but here's my 2-cents' worth about it:
The instructional-design world is full of evaluation methods. And the business world has metrics galore that can be analyzed before and after training.
IMHO, the first step is needs analysis, which should occur before anything is developed. What is needed to meet the business goal? Because if you don't know the goal, how can you measure success? The best info I've seen on needs analysis comes from Cathy Moore: https://blog.cathy-moore.com/2013/05/is-training-really-the-answer-ask-the-flowchart/
As I mentioned above, to do a proper evaluation, you need to know what "effective" looks like. That typically means gathering metrics before and after the training. Unfortunately, there may not be direct, measurable metrics, especially with compliance training. That's why the fallback measurement tends to be "Did the user pass the test?"
I wish I had more helpful input. Perhaps another ID in the Forum who has more experience with evals will chime in.
Training requirements in regulations usually focus on information delivery, not application, so part of the problem we face with regulatory/compliance courses is the lousy nature of the requirements themselves. We desperately need regulations that encapsulate a better understanding of what constitutes effective training.
But, we don't have that. So the next best thing we can do is try to figure out what the spirit of the regulations is and not get complacent that it's enough to just follow the literal requirements. Usually, we want people to know something for a reason, not just because some regulation says we have to tell them. What is that reason? What do we think the regulators were trying to protect people from or alert people to and why do we think they felt it was important enough to make it a legal requirement in the first place?
Most of the time, the reason we want people to know something is because we want them ultimately to ACT on that information. The information must inform an action or a decision, otherwise it can have little or no impact on the real world. Knowledge you don't use is no more useful than knowledge you don't have.
Therefore, the best way to test for knowledge/understanding in any course--even in a compliance course--is to put learners in situations that require them to make decisions based upon the information they have learned. Can they make the right decision? Can they articulate why? Then they have understood the material.
My group is focused on safety training, a lot of which is driven by regulatory compliance, but we always try to put the material in the context of how it would be used by someone in the target audience for the course. We gave a talk last year and had to throw together some examples to illustrate our approach. Here is the "highlight reel" we shared (there are a few issues with it because we cobbled this together by lifting excerpts from a bunch of separate courses at the last minute, but it still gives a pretty good idea of what we are trying to achieve and how we go about it).
If you spend some time in this highlight reel, you'll see that these examples are very different from the ordinary approach to compliance training. There is much more focus on job context and application, and much less focus on facts and remembering policy.
Many pages in the highlight reel have an About button that tries to explain a little about what is interesting or what we were trying to do in that example. There are Back buttons on most pages that let you get quickly back to the main menu.
Thanks for sharing this Ray. This is great information and example I can share with my team. I love that in the scenario you labeled photography, the learner had to also give a reason for the choice they select, making them think about the why. I haven't gone through all examples but this is great.
Just a quick follow up to let you know that there is a free online conference completely devoted to assessments coming up November 3-5 of this year. This morning I registered to attend. Not sure how much of the conference I'll actually be able to watch in realtime, but it's free and I'm interested in hearing Clark Aldrich's talk, at least.
Anyway, here's the website, in case you are interested.
8 Replies
I've done a lot of compliance courses. Those are often made to provide proof that someone has received info about a specific policy, procedure, etc.
Evaluating training effectiveness is a whole other topic. For example, how would you define "effective" if the main goal is just to present info for compliance purposes? Plus, training is only one of many factors that impact whether someone actually follows a procedure. So people passing a quiz in an eLearning course might prove compliance with regulations re: providing info, but it can't prove the effectiveness of that info.
Thanks Judy, you raise really good points which have sparked additional questions for me to take back. For example, what is important to the client - are they looking to check a box that the training launched and that people completed?
Regarding the training effectiveness, with your experience developing compliance courses, have you had to evaluate and what do you think works, or what would you recommend.
Thanks for sharing.
Hi again, Audrey,
I haven't been involved in training-effectiveness evaluations, but here's my 2-cents' worth about it:
The instructional-design world is full of evaluation methods. And the business world has metrics galore that can be analyzed before and after training.
IMHO, the first step is needs analysis, which should occur before anything is developed. What is needed to meet the business goal? Because if you don't know the goal, how can you measure success? The best info I've seen on needs analysis comes from Cathy Moore: https://blog.cathy-moore.com/2013/05/is-training-really-the-answer-ask-the-flowchart/
As I mentioned above, to do a proper evaluation, you need to know what "effective" looks like. That typically means gathering metrics before and after the training. Unfortunately, there may not be direct, measurable metrics, especially with compliance training. That's why the fallback measurement tends to be "Did the user pass the test?"
I wish I had more helpful input. Perhaps another ID in the Forum who has more experience with evals will chime in.
Thanks Judy! Much appreciated.
Training requirements in regulations usually focus on information delivery, not application, so part of the problem we face with regulatory/compliance courses is the lousy nature of the requirements themselves. We desperately need regulations that encapsulate a better understanding of what constitutes effective training.
But, we don't have that. So the next best thing we can do is try to figure out what the spirit of the regulations is and not get complacent that it's enough to just follow the literal requirements. Usually, we want people to know something for a reason, not just because some regulation says we have to tell them. What is that reason? What do we think the regulators were trying to protect people from or alert people to and why do we think they felt it was important enough to make it a legal requirement in the first place?
Most of the time, the reason we want people to know something is because we want them ultimately to ACT on that information. The information must inform an action or a decision, otherwise it can have little or no impact on the real world. Knowledge you don't use is no more useful than knowledge you don't have.
Therefore, the best way to test for knowledge/understanding in any course--even in a compliance course--is to put learners in situations that require them to make decisions based upon the information they have learned. Can they make the right decision? Can they articulate why? Then they have understood the material.
My group is focused on safety training, a lot of which is driven by regulatory compliance, but we always try to put the material in the context of how it would be used by someone in the target audience for the course. We gave a talk last year and had to throw together some examples to illustrate our approach. Here is the "highlight reel" we shared (there are a few issues with it because we cobbled this together by lifting excerpts from a bunch of separate courses at the last minute, but it still gives a pretty good idea of what we are trying to achieve and how we go about it).
https://training.lbl.gov/ehs/training/webdev/LBNL_EFCOG/
If you spend some time in this highlight reel, you'll see that these examples are very different from the ordinary approach to compliance training. There is much more focus on job context and application, and much less focus on facts and remembering policy.
Many pages in the highlight reel have an About button that tries to explain a little about what is interesting or what we were trying to do in that example. There are Back buttons on most pages that let you get quickly back to the main menu.
Thanks for sharing this Ray. This is great information and example I can share with my team. I love that in the scenario you labeled photography, the learner had to also give a reason for the choice they select, making them think about the why. I haven't gone through all examples but this is great.
Thank you very much for sharing.
Just a quick follow up to let you know that there is a free online conference completely devoted to assessments coming up November 3-5 of this year. This morning I registered to attend. Not sure how much of the conference I'll actually be able to watch in realtime, but it's free and I'm interested in hearing Clark Aldrich's talk, at least.
Anyway, here's the website, in case you are interested.
http://beyond-multiple-choice.com/
Cheers!
This is great! Thanks for sharing :)
This discussion is closed. You can start a new discussion or contact Articulate Support.