After reading an article Tweeted by Articulate this morning re: training KPIs, I would be keen to hear thoughts around Compliance KPIs and how we can actually measure the effectiveness of compliance training.
Largely, I feel that analysis needs to take place over longer periods of time and requires greater (and longer-term) investment from both the provider and the client. Does anyone here have any direct experience of this?
Hi Helen. I use two ways for that depending on the situation:
1. A follow-up mini-module about 6 weeks after. It's mainly a memory-jogger/quiz format, but with no new content introduced. Wrongly answered questions trigger sub-branches pointing back to the original module/workbook. There's sort of a self-reflection score at the end, i.e. scoring less than x% and it 'strongly suggests' the learner might be putting themselves/their colleagues/their organisation at risk since they were so far off the mark.....so maybe they should consider retaking the original module again.
or: 2. I put a 1 page .pdf assignment in the player resources tab. The validation is more workplace-based as it promts them to go off and do something practical but related to the theory, and then write it up in a simple way. That's then e-mailed to an SME or a line manager (or me) for review who can take appropriate action.
Since I work for a rather large and complex organization with different departments with different requirements, we first identified the beneficiaries of the course, that is, who is most impacted by the given training. From that point on we recommend the appropriate approach based on their specific objectives.
We found that there are several approaches that are more effective in certain situations than others, so we use different methods. One of them is the 2-2-2 model of reinforcing (inspired by Art Kohn) with different types of interactions that the beneficiaries find more suitable (i.e., email messages, text messages -there's a company that actually offers follow-up services by text, interactive questions, etc). The 2-2-2 is sending the booster(s) 2 days, 2 weeks and 2 months after the learning event.
Additionally, we determine what type of recalling/memorizing/hands-on concept application will be sent in the booster. The idea is to make the learner think and recall back, generate an idea or demonstrate integration of what was learned, so we try to stimulate the memory or have the learner integrate the application of a given concept.
I would add set up a compliance calendar so that people understand their is a requirement to complete compliance training.
Its good to use stats from other departments on a three month time period once the training is complete. So if it was health and safety number of accidents or near misses, Occ health, back injuries, customer service number of complaints /praises. You can then build up a yearly picture. Sometimes a set of questions three months after course. Compliance training is often called refresher as their are often a few new items to learn.
You should find compliance training making a reduction in these KPI's.
I particularly like the idea of sending a follow-up module that implicates the learner if they do not achieve a pass rate. Through putting the responsibility of compliance on their shoulders by showing that they may be letting their coworkers down is a clever trick. I don't suppose you draw data from these follow up exercises to see how many people fail the mini module and then compare it to how many people re-take the full course?
I also really like the idea of follow up, independent activities. I presume this requires close-working with the SME to determine how reminders to complete these activities are sent out/whether they can answer them etc.?
Using boosters - whether mini-courses, emails or texts, seems like a great way to check individuals are still able to understand what is needed of them in order to comply. It would be great to make this into a type of game, based on real life situations, i.e. you receive an alert about possible misconduct, and you must answer a question related to what you think the best approach is. The players could rack up scores, and there could be a compliance leader board, perhaps. In effect, these mini activities would just be reinforcing the main learning objectives of a course, but making recall more interesting/relevant....
Some real thought-provoking stuff here - thanks again,
Would a compliance calendar be LMS based? Or were you thinking of something more tangible?
Determining exactly what stats can be measured as a direct result of the learning is something I'm concerned with, and something that is tricky with particular compliance topics such as AML/ABC/Code of Conduct etc., especially when clients often deliver learning to blanket audiences.
Comparing stats across departments would be a terrific way to instill competition and make people more aware that their levels of compliance would actually have an effect on other colleagues, across the company.
Yes, that approach works a treat, in my experience; e.g. I once did a branched scenario game with sequences of video clips of stakeholders that gave a interactive story feeling based on how the user clicked. If you remember 'Dungeons and Dragons'....I based it on that.
It was on disaster recovery, and basically simulated the decisions made in an emergency scenario. Various parties got increasingly more irate. It was great fun!
If you can get the lms to trigger at certain points to ask specific questions is the best solution. It gives you a better overall picture - if you don't have an lmd try and get one. Its a bit like expecting an account to do their job with no access to excel.
Some have a graph system so you can see the output so its easier to analyse.
I have also posted this on another question but you can't really beat the director of compliance or the CEO having a short 90 second video at the start of the training to explain the importance of the training. Its also a bit of trick as people think the training is coming from the cel so the completion is quicker.
The compliance calendar kept you right and to be honest in a heavily audited industries these tend to relax auditors. The benefits are spread far a wide and not just another compliance training module.
I also found a 60-90 second video of a compliance manager highlighting a key point can break up the module and offer a human interaction.
I always think compliance training is a great way to unleash your inner artist and just throw in lots of ideas and concepts to make the training interesting.
Also sometimes its good to start with what happens when compliance goes wrong. Everyone no doubt everyone has a mortgage and its good to highlight negative impacts of what happens when compliance training isn't completed. The scare tactic is more a show the two sides of a coin (good audit or bad audit).
Hi Helen, when we conducted our Analysis prior to the last compliance course build we id'd 2 things that you might be able to use - The first was that the regulation was really all about "doing the right thing"; that is, to say, that the actions the learners are expected to take are the ones that are ethically and morally right. We knew from other analysis that those same people were already motivated to do what's right; but where it got sticky for them was that their idea of "right" wasn't necessarily the same as the regulator's idea of "right". We discussed this A LOT with the SME's, and determined that in the end the message we wanted the learners to walk away with was 2 fold - 1: do what's right. 2: if you're not sure if it's right, ask someone. So, we built all of our scenarios around getting the learners to determine whether or not the action to take was, indeed, the right one from the regulators pov.
The second speaks to your measurement question - the SME's had a hotline that learners were expected to call if they had questions. We knew how many calls were being received prior to the launch, and predicted that the volume would increase after the course was deployed. So we used that to determine ROI/success, and will eventually evaluate the "new" call data for course revision information.
11 Replies
Hi Helen. I use two ways for that depending on the situation:
1. A follow-up mini-module about 6 weeks after. It's mainly a memory-jogger/quiz format, but with no new content introduced. Wrongly answered questions trigger sub-branches pointing back to the original module/workbook. There's sort of a self-reflection score at the end, i.e. scoring less than x% and it 'strongly suggests' the learner might be putting themselves/their colleagues/their organisation at risk since they were so far off the mark.....so maybe they should consider retaking the original module again.
or:
2. I put a 1 page .pdf assignment in the player resources tab. The validation is more workplace-based as it promts them to go off and do something practical but related to the theory, and then write it up in a simple way. That's then e-mailed to an SME or a line manager (or me) for review who can take appropriate action.
Since I work for a rather large and complex organization with different departments with different requirements, we first identified the beneficiaries of the course, that is, who is most impacted by the given training. From that point on we recommend the appropriate approach based on their specific objectives.
We found that there are several approaches that are more effective in certain situations than others, so we use different methods. One of them is the 2-2-2 model of reinforcing (inspired by Art Kohn) with different types of interactions that the beneficiaries find more suitable (i.e., email messages, text messages -there's a company that actually offers follow-up services by text, interactive questions, etc). The 2-2-2 is sending the booster(s) 2 days, 2 weeks and 2 months after the learning event.
Additionally, we determine what type of recalling/memorizing/hands-on concept application will be sent in the booster. The idea is to make the learner think and recall back, generate an idea or demonstrate integration of what was learned, so we try to stimulate the memory or have the learner integrate the application of a given concept.
I would add set up a compliance calendar so that people understand their is a requirement to complete compliance training.
Its good to use stats from other departments on a three month time period once the training is complete. So if it was health and safety number of accidents or near misses, Occ health, back injuries, customer service number of complaints /praises. You can then build up a yearly picture. Sometimes a set of questions three months after course. Compliance training is often called refresher as their are often a few new items to learn.
You should find compliance training making a reduction in these KPI's.
Hi Mike,
Thanks for the reply.
I particularly like the idea of sending a follow-up module that implicates the learner if they do not achieve a pass rate. Through putting the responsibility of compliance on their shoulders by showing that they may be letting their coworkers down is a clever trick. I don't suppose you draw data from these follow up exercises to see how many people fail the mini module and then compare it to how many people re-take the full course?
I also really like the idea of follow up, independent activities. I presume this requires close-working with the SME to determine how reminders to complete these activities are sent out/whether they can answer them etc.?
Thanks again,
Helen
Hi Ulises,
Thanks for taking the time to reply.
Using boosters - whether mini-courses, emails or texts, seems like a great way to check individuals are still able to understand what is needed of them in order to comply. It would be great to make this into a type of game, based on real life situations, i.e. you receive an alert about possible misconduct, and you must answer a question related to what you think the best approach is. The players could rack up scores, and there could be a compliance leader board, perhaps. In effect, these mini activities would just be reinforcing the main learning objectives of a course, but making recall more interesting/relevant....
Some real thought-provoking stuff here - thanks again,
Helen
Hi Malcolm,
Thanks for responding.
Would a compliance calendar be LMS based? Or were you thinking of something more tangible?
Determining exactly what stats can be measured as a direct result of the learning is something I'm concerned with, and something that is tricky with particular compliance topics such as AML/ABC/Code of Conduct etc., especially when clients often deliver learning to blanket audiences.
Comparing stats across departments would be a terrific way to instill competition and make people more aware that their levels of compliance would actually have an effect on other colleagues, across the company.
Thanks again,
Helen
No, not found the need to data-mine that far as it's been sufficient for one round of reminders, in the stuff I've done.
Yes, sort of.....part of the ID/Storyboarding stage.
:-)
Yes, that approach works a treat, in my experience; e.g. I once did a branched scenario game with sequences of video clips of stakeholders that gave a interactive story feeling based on how the user clicked. If you remember 'Dungeons and Dragons'....I based it on that.
It was on disaster recovery, and basically simulated the decisions made in an emergency scenario. Various parties got increasingly more irate. It was great fun!
Video branching is something I would love to do one day! I think this type of learning is extremely effective and really valuable.
Thanks for replying!
Hi Helen
If you can get the lms to trigger at certain points to ask specific questions is the best solution. It gives you a better overall picture - if you don't have an lmd try and get one. Its a bit like expecting an account to do their job with no access to excel.
Some have a graph system so you can see the output so its easier to analyse.
I have also posted this on another question but you can't really beat the director of compliance or the CEO having a short 90 second video at the start of the training to explain the importance of the training. Its also a bit of trick as people think the training is coming from the cel so the completion is quicker.
The compliance calendar kept you right and to be honest in a heavily audited industries these tend to relax auditors. The benefits are spread far a wide and not just another compliance training module.
I also found a 60-90 second video of a compliance manager highlighting a key point can break up the module and offer a human interaction.
I always think compliance training is a great way to unleash your inner artist and just throw in lots of ideas and concepts to make the training interesting.
Also sometimes its good to start with what happens when compliance goes wrong. Everyone no doubt everyone has a mortgage and its good to highlight negative impacts of what happens when compliance training isn't completed. The scare tactic is more a show the two sides of a coin (good audit or bad audit).
Hope this helps
Malcolm
Hi Helen, when we conducted our Analysis prior to the last compliance course build we id'd 2 things that you might be able to use - The first was that the regulation was really all about "doing the right thing"; that is, to say, that the actions the learners are expected to take are the ones that are ethically and morally right. We knew from other analysis that those same people were already motivated to do what's right; but where it got sticky for them was that their idea of "right" wasn't necessarily the same as the regulator's idea of "right". We discussed this A LOT with the SME's, and determined that in the end the message we wanted the learners to walk away with was 2 fold - 1: do what's right. 2: if you're not sure if it's right, ask someone. So, we built all of our scenarios around getting the learners to determine whether or not the action to take was, indeed, the right one from the regulators pov.
The second speaks to your measurement question - the SME's had a hotline that learners were expected to call if they had questions. We knew how many calls were being received prior to the launch, and predicted that the volume would increase after the course was deployed. So we used that to determine ROI/success, and will eventually evaluate the "new" call data for course revision information.
Hope that helps.
This discussion is closed. You can start a new discussion or contact Articulate Support.