Can anyone please point me to an article or research that supports 80% pass rate as good/normal practice? I'm trying to make a case for reducing our pass rates from 100%.
I usually try and get the conversation going by asking clients which 20% of the course they want me to scrap as it's obviously not important ;)
I appreciate this over-simplifies things a little, but it starts the conversation going. I think 100% is a perfectly reasonable target in some circumstances, the important thing is that it is a realistic and logical % pass rate, and not just dogma.
This is a super intriguing topic to me! I would love to hear from others in the community about how they decide what to set the passing score as. I think in many cases it might just be an arbitrary choice, but in the cases where it isn't I'd love to hear how people arrived to the their final score.
One thing about the 100% pass rate is that, as a learner, if I know I need to get 100% to pass the course (and I think it's a best practice to let learners know before they start a quiz how many questions there will be and what the pass rate is) as soon as I get a wrong answer, I know I've failed the quiz. That might discourage me or make me think "what's the point?" to answering the other questions since I know I've failed already. Obvi the same logic could apply if the pass score is 75% or 85%, doesn't matter, but when it's 100% it's so clear that as soon as I get even one wrong, I've failed. Just something to consider. For that reason, I might give a bit of leeway and set a score at 95%.
Louise, just wondering, why do you want to reduce the pass rates from 100% to 80% ? I'm sure there are some specific reasons or a situation that is making you consider this? I'm very interested!
"Everything in this course is information that you need to do your job well, and as such, we want you to know everything you've been taught. If you don't get it all right, all you need to do is go back and have another look at the sections you don't know, and then try again later...", (or similar).
"Quizzes" are often so "combative". Make the audience WANT to get 100% by making it all RELEVANT. This requires so much more than just throwing eLearning at them, and the eLearning needs to be tied to the business, and their day-to-day reality rather than just being thrown at them to solve some unexplained, irrelevant (to them...) problem.
Also - do not show individual scores, show at the end. That takes away the "Why should I bother going on...?" problem.
Agree with Bruce on the 100%. If it's OK to get 20% wrong, that 20% could be any of the 100%. By making the cut score an arbitrary 80% of the whole, we're rendering any of the quiz challenges as potentially irrelevant.
The tension created by "people make mistakes" and not wanting to put folks into a loop is exacerbated by tools that don't make it that easy to selectively remediate based on a particular answer. In many cases, the natural function of the tools we use to build often requires participants to take the entire assessment again, even the parts that they've already demonstrated mastery.
So we fall back on a lower artibrary / reasonable passing score to compensate and avoid breeding more animosity in our audience. We do this without regard to the circumstance or values of individual questions or challenges.
Angoff, Nedelsky, and Ebel's methods each rely on making a statistical judgment for each question and choice. This is certainly better than an abitrary overall pass rate. In my last job we prescribed the Angoff method for high stakes assessments.
Most elearning assessments aren't high stakes. But I'd still contend that if it was important enough to design, build and to ask for participation, isn't it important enough to make sure folks "got it" 100%? And if not, it's probably time to start trimming the learning experience so that the 20% that's not relevant can be cut, as Bruce points out:)
Breaking the assessments up into smaller sections also helps a bit. Failing a very large examination is disheartening when you know you're going to have to retake the entire thing. Missing one question in a chain of 3 is less destructive to the spirit than missing three in a chain of 20.
Storyline 2 improves the situation a bit by allowing partial credit. With work, you can allow folks to miss questions and direct them to remedial activities for extra attention. We've done this a few times with a pre-course quiz that tailored the pass requirements based on quiz performance.
Hi all - thanks for the great comments! Certainly given me a new perspective.
The courses are compliance courses and I'm reviewing the pass rates as at present some courses are 100% and others are 80%. Learner and Manager feedback is that 100% is disheartening. Whichever way we decide to go I feel all courses should have same pass rate otherwise it would indicate a 100% course is more important?
Here's a different version of my thoughts - I think many of the people that commission and design eLearning are just plain scared, scared of actually being held to account for measurable business success,
Sorry, I do not "get" the whole "...they might be discouraged" thing. Success is what we aim for, in any endeavor, and certainly in business. Change the game - and encourage them to succeed! Imagine what would be said in the boardroom and by shareholders if the Annual Report said "It's OK, we hit 80% of our forecast" Imagine what would happen to R&D if on a product launch date they said "It's OK, 80% of our product was launched". Salespeople do not get comped on hitting 80% target - they often get fired.
Why do L&D ALWAYS think they are different? It's not surprising that many businesses/boards do not take L&D/HR seriously when we perpetuate this fluffy nonsense.
Our job is to do it RIGHT - be focused on what is correct for the business, not panda to "political correctness" and people that are scared of failure. So many IDs just don't get this, so many companies just don't get this. Challenge people to get it right. Set the expectations more appropriately.
The businesses I have worked in have been hard/harsh environments - there's nothing wrong with that. That's why they were individually so successful. We did not do what was expected, we did what was right.
Not sure about your VPs and Presidents, but none of mine EVER accepted that a 20% failure rate was OK. They wanted 100% success, they got it (one way or another), and the business results showed that.
Forget 80%, aim for 100%, and find a way to make it happen that makes sense. It might mean you have to work a little harder designing your course, but whoever said the job of an ID was easy.
It really depends on the type of course and why you need an assessment. Have you ensured that the assessment accurately measures the content of the course.
If it does and it is a compliance/mandatory course then 100% correct is good. But I would always ask the question "Do you really need a quiz?". Are there other ways you can measure that they have taken the course? Could you measure something in real life based on what they have learnt.
My real concern with quizzes/assessments in courses is that you are only measuring their retention of information after taking the course, which although not measuring short-term memory, you are only measuring their performance at that point in time.
Most elearning courses I have built have been looking for behavioural changes and a quiz after a course will not measure that, I find a lot of assessments are used in elearning because it is very easy to run a report of the LMS to show who passed/failed rather than look at measuring the actual reason for building the course in the first place.
@Phil - agreed, (you know my thoughts around this topic... :)
Even with compliance courses there are much better ways to do it than "Quizzes". It's one area where I feel the eLearning industry does not do itself any favours in gaining acceptance within the bigger business picture.
@Louise - no, not planing to, have rather a lot on one way or another for about the next 2 months! :)
Maybe "should we lower the cut score?" is the wrong question. Maybe "how can we improve the experience and still aim for 100%?" is a better question. Why do folks indicate that 100% is disheartening? Fixing those specific problems might be a good place to focus.
Here are some potential areas folks might indicate as problems:
"I take this long assessment and if I miss a few questions (for whatever reason) I need to take the whole assessment over again. That sucks."
"Some of the questions seem tricky, where a choice could be right given the right circumstances. Why do I need to take the entire thing over again for stuff like that?"
"Many of the questions ask about information that really doesn't matter? Why should I care what section of the policy states ________.? I won't remember it in a week anyway."
There are lots of ways to improve the design of an assessment and the mechanisms used to deliver both the assessment and the feedback. Perhaps setting up a few suggested cases to adjust the model could help to make the case for keeping the cut where it should be (100%) while improving the overall experience of taking the assessment, particularly when questions are missed?
It would be nicer to build a course that is shortened if they already know what we are trying to tell you.
I envisage a course where you are asked questions throughout and if you get them right you almost do the speed readers version, if you get them wrong you are led through the process.
The assessment doesn't score you but instead chooses the content you need to view.
That's a good point, Phil. Elearning courses should probably never stand by themselves. Lots of compliance courses end up as a legal checkmark. It's convenience, I suppose:) But a training solution should want to change behavior, not cover an organization legally. Legal bulwark is something else entirely.
I've proposed that we have a series of legal acknowledgements for folks to read and indicate agreement as a foundation for values / legal issues. Rolling these things up in a training package masks the intent.
All these old habits we have to break and change:)
Love the concept of a tailoring assessment or a conversational component that helps the solution adapt to the needs / situation of the participant. Instead of the other way around:)
Love the concept of a tailoring assessment or a conversational component that helps the solution adapt to the needs / situation of the participant. Instead of the other way around:)
But Steve...that would cost money/time to do well! Perish the thought.
Maybe a little. But there are degrees of effort:) Take this model for example:
Topic Menu Screen
Topic 1
Topic 2
Topic 3
Topic 4
Assessment for Topics 1 - 4
Shift it to this, where the assessment is graded by topic and the Topic Menu indicates completion by topic after the assessment is completed. The sections folks need to go through are clearly marked (tailored) by the assessment. Someone who already knows the info doesn't need to take the course at all. Someone who knows half, gets to take half. A little bit of additional effort to make this happen but it's largely a re-order. The side-benefit of providing those who already know the information an easy way to complete the module.
I just built a course, where you are guided through using a software tool. There are 4 sections and in each section you are asked questions (where do you press to launch...?) if you get any question wrong you are guided to a coach (the learning) who takes you through the process. This way you pass the assessments you never see the learning, if you fail you a question you do the learning. This in effect asks for 100% pass rate or else you do the learning.
This whole course is has a wrapper that makes it look like a cartoon and also gives rewards as you progress through the course. The biggest reward however is if you get 100% on each section you do not need to view the learning and spend less time.
I am loving this conversation, a lot of good points have been made! Thanks so much Bruce, Phil, and Steve for heeding the call and participating, love to hear your thoughts and opinions on this matter.
Thinking of my own past educational experiences (university, college, some online training, etc), interestingly, I never had to meet a passing score of 100%. Some very of good points for the 100% pass score! I suppose that, in the end, even if the learner gets 100% it doesn't "prove" much since we never know which answers were a lucky guess, and since quizzing typically only covers a sample of the material, not all.
The great thing here Louise is there is NO right or wrong answer to "what should a passing score be?". We all have to make decicions, as instructional designers, based on the variables and constraints of our own projects, so there's no one right answer. As pointed out in that handy reference link, at the end of the day, the passing score is a judgement call/opinion and as long as you can articulate solid reasoning for the score you've selected, based on your situation, which your boss/stakeholder agrees with, then I think you're headed in the right direction!
In your current situation, you might not have the option to re-design the courses. And if feedback indicates that, as they are, learners think the 100% pass is disheartening, it might make the most sense for now to lower the passing score to 80% and make it consistent across the board. Might be a good idea to schedule a meeting with management to discuss some of the business points brought up by Bruce, Phil, and Steve so that you can design your future courses in a way that best lets you incorporate a 100% pass rate, if you decide to go that way! :) Thanks again for bringing up this great topic, Louise!
I totally agree with you Nicole! There really is no "right" answer here, it is just a judgement call.
I like what David Glow said in this discussion about changing the culture of assessment so that scores become irrelevant and focus on providing feedback that actually helps the learner to better understand the concepts. However, I understand that in the context of corporate training, especially compliance training, that is not always possible.
I see what Bruce and Phil are saying about what setting the passing score 80% implies (that 20% of the content is not important) but at the same time, Nicole is right: in school we never had to get 100% to pass! One of the reasons getting a score of 100% may be "disheartening" to some is because the questions may be confusing or have several plausible answers.
For these reasons, I'm going to have to side with Nicole on this. It seems like in your case it would make sense to lower the score of existing courses to 80% and then think about possibly changing the way you build courses from here on out in order to try and take into account some of the ideas presented here.
Not sure if I ever came down agreeing on 100%. I would argue that an assessment in courses that are looking at behavioural changes of employees an assessment is a poor way of measuring this change and you should ideally look at your business metrics to see if there is a better way of measuring this.
I would recommend designing your courses differently so the user gets 100% but never actually knew they took an assessment, you would tailor the learning based on how they answer the questions and ensure that they get it correct before they are allowed to finish the course.
I really do not like the idea of assessments just for compliance sake and would always recommend doing it another way.
"Quizzes" are supposed to be a formative assessment (practice stuff). What I like to do is intersperse the "quiz" type of questions while showing how they help our employees improve their performance on the job. The summative assessments are questions on how the new learning is applied to their job role... so 100% is the norm, if anything is "missed" then feedback gives corrective action and the applied scenario is given again but in a different example. I would agree that if it's ok to miss 20% of the questions, let's take out 20% of the content to reduce cognitive overload and get 100% of whatever is left :o)
I find the hard part is for everyone to agree on what is need-to-know (even in a compliance course), but once we have that figured out, I take out the nice-to-knows and focus on what is necessary to improve job performance. I do not add anything to courses that is fluff or it'd be be neat if they knew this.
It is pretty easy to do this, I ask: give me an example of when this bit of information is USED during their normal daily tasks, if they can't, it's out. If they can, then we focus on 100% learning.
What we have to remember is that "learning" is when we change people's behavior. If they can't use the topics/concepts you are showing them, it's just nice to know, then there is no learning. But if there is learning to be done, we strive for 100%. It's ID mindset, it's management mindset, it's a work culture mind set.
I am really enjoying the discussion in this post. My question is somewhat related which is how much push back is too much? At some point, whether it's good ID theory or not, when is it acceptable to give the customer what they want? The only reason I ask, is that I was involved in building compliance courses and the customers wanted everything in the "what not to do" category, (locked slides, text and audio, no scenarios or interactvitiy, Quiz with 80%) just forced "legal" learning. And, as much as I pushed, I finally felt I had to give in and just do what "they" wanted. Would love to hear your thoughts? Or maybe this is a different post for a different day. :)
That's always the rub. Folks have expectations and, largely, those expectations are that content is designed for their preferences and what they're used to and not what's good for the business requirements or for the participant.
Like any change, it really takes time. And folks need to see something new actually work before they buy into it. It's tough. I end up pushing back a bit on just about every project. I win a little, I lose a little. I usually give quite a bit in both compromise as well as going above and beyond expectations to show how things could be a little better. Making stuff suck a just little bit less over time. The conversation around business requirements and respect for the individuals participating in the experience is what matters. It's never a fight in a conflict sense, but selling something different is hard work. Like all selling:) First they have to know they need it.
It'll work over time. Expectations will shift little by little until (someday far into the future) we're all chuckling over how bad things were 20 years ago. The long game:)
29 Replies
Hi Louise!
I don't know of an article that supports 80% as a pass rate, however I have often referred to this (Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests) when looking for information on how to decide on a passing score... hope this helps a bit and hope we hear from others in the community on this!
Interesting topic... people do often seem to go with the 80% pass rate and it's an interesting discussion for sure.
Thanks for posting your question here in the E-Learning Heroes forum! :)
Hi Louise!
I usually try and get the conversation going by asking clients which 20% of the course they want me to scrap as it's obviously not important ;)
I appreciate this over-simplifies things a little, but it starts the conversation going. I think 100% is a perfectly reasonable target in some circumstances, the important thing is that it is a realistic and logical % pass rate, and not just dogma.
Very interesting points, Bruce.
This is a super intriguing topic to me! I would love to hear from others in the community about how they decide what to set the passing score as. I think in many cases it might just be an arbitrary choice, but in the cases where it isn't I'd love to hear how people arrived to the their final score.
One thing about the 100% pass rate is that, as a learner, if I know I need to get 100% to pass the course (and I think it's a best practice to let learners know before they start a quiz how many questions there will be and what the pass rate is) as soon as I get a wrong answer, I know I've failed the quiz. That might discourage me or make me think "what's the point?" to answering the other questions since I know I've failed already. Obvi the same logic could apply if the pass score is 75% or 85%, doesn't matter, but when it's 100% it's so clear that as soon as I get even one wrong, I've failed. Just something to consider. For that reason, I might give a bit of leeway and set a score at 95%.
Louise, just wondering, why do you want to reduce the pass rates from 100% to 80% ? I'm sure there are some specific reasons or a situation that is making you consider this? I'm very interested!
Why not try re-framing it somewhat?
"Everything in this course is information that you need to do your job well, and as such, we want you to know everything you've been taught. If you don't get it all right, all you need to do is go back and have another look at the sections you don't know, and then try again later...", (or similar).
"Quizzes" are often so "combative". Make the audience WANT to get 100% by making it all RELEVANT. This requires so much more than just throwing eLearning at them, and the eLearning needs to be tied to the business, and their day-to-day reality rather than just being thrown at them to solve some unexplained, irrelevant (to them...) problem.
Also - do not show individual scores, show at the end. That takes away the "Why should I bother going on...?" problem.
Agree with Bruce on the 100%. If it's OK to get 20% wrong, that 20% could be any of the 100%. By making the cut score an arbitrary 80% of the whole, we're rendering any of the quiz challenges as potentially irrelevant.
The tension created by "people make mistakes" and not wanting to put folks into a loop is exacerbated by tools that don't make it that easy to selectively remediate based on a particular answer. In many cases, the natural function of the tools we use to build often requires participants to take the entire assessment again, even the parts that they've already demonstrated mastery.
So we fall back on a lower artibrary / reasonable passing score to compensate and avoid breeding more animosity in our audience. We do this without regard to the circumstance or values of individual questions or challenges.
Angoff, Nedelsky, and Ebel's methods each rely on making a statistical judgment for each question and choice. This is certainly better than an abitrary overall pass rate. In my last job we prescribed the Angoff method for high stakes assessments.
Most elearning assessments aren't high stakes. But I'd still contend that if it was important enough to design, build and to ask for participation, isn't it important enough to make sure folks "got it" 100%? And if not, it's probably time to start trimming the learning experience so that the 20% that's not relevant can be cut, as Bruce points out:)
Breaking the assessments up into smaller sections also helps a bit. Failing a very large examination is disheartening when you know you're going to have to retake the entire thing. Missing one question in a chain of 3 is less destructive to the spirit than missing three in a chain of 20.
Storyline 2 improves the situation a bit by allowing partial credit. With work, you can allow folks to miss questions and direct them to remedial activities for extra attention. We've done this a few times with a pre-course quiz that tailored the pass requirements based on quiz performance.
Hi all - thanks for the great comments! Certainly given me a new perspective.
The courses are compliance courses and I'm reviewing the pass rates as at present some courses are 100% and others are 80%. Learner and Manager feedback is that 100% is disheartening. Whichever way we decide to go I feel all courses should have same pass rate otherwise it would indicate a 100% course is more important?
Would love to hear other views on this.
Bruce - are you going to Olympia this year?
Louise
Thanks for that Steve.
Here's a different version of my thoughts - I think many of the people that commission and design eLearning are just plain scared, scared of actually being held to account for measurable business success,
Sorry, I do not "get" the whole "...they might be discouraged" thing. Success is what we aim for, in any endeavor, and certainly in business. Change the game - and encourage them to succeed! Imagine what would be said in the boardroom and by shareholders if the Annual Report said "It's OK, we hit 80% of our forecast" Imagine what would happen to R&D if on a product launch date they said "It's OK, 80% of our product was launched". Salespeople do not get comped on hitting 80% target - they often get fired.
Why do L&D ALWAYS think they are different? It's not surprising that many businesses/boards do not take L&D/HR seriously when we perpetuate this fluffy nonsense.
Our job is to do it RIGHT - be focused on what is correct for the business, not panda to "political correctness" and people that are scared of failure. So many IDs just don't get this, so many companies just don't get this. Challenge people to get it right. Set the expectations more appropriately.
The businesses I have worked in have been hard/harsh environments - there's nothing wrong with that. That's why they were individually so successful. We did not do what was expected, we did what was right.
Not sure about your VPs and Presidents, but none of mine EVER accepted that a 20% failure rate was OK. They wanted 100% success, they got it (one way or another), and the business results showed that.
Forget 80%, aim for 100%, and find a way to make it happen that makes sense. It might mean you have to work a little harder designing your course, but whoever said the job of an ID was easy.
Just my 2p worth.
It really depends on the type of course and why you need an assessment. Have you ensured that the assessment accurately measures the content of the course.
If it does and it is a compliance/mandatory course then 100% correct is good. But I would always ask the question "Do you really need a quiz?". Are there other ways you can measure that they have taken the course? Could you measure something in real life based on what they have learnt.
My real concern with quizzes/assessments in courses is that you are only measuring their retention of information after taking the course, which although not measuring short-term memory, you are only measuring their performance at that point in time.
Most elearning courses I have built have been looking for behavioural changes and a quiz after a course will not measure that, I find a lot of assessments are used in elearning because it is very easy to run a report of the LMS to show who passed/failed rather than look at measuring the actual reason for building the course in the first place.
@Phil - agreed, (you know my thoughts around this topic... :)
Even with compliance courses there are much better ways to do it than "Quizzes". It's one area where I feel the eLearning industry does not do itself any favours in gaining acceptance within the bigger business picture.
@Louise - no, not planing to, have rather a lot on one way or another for about the next 2 months! :)
Maybe "should we lower the cut score?" is the wrong question. Maybe "how can we improve the experience and still aim for 100%?" is a better question. Why do folks indicate that 100% is disheartening? Fixing those specific problems might be a good place to focus.
Here are some potential areas folks might indicate as problems:
There are lots of ways to improve the design of an assessment and the mechanisms used to deliver both the assessment and the feedback. Perhaps setting up a few suggested cases to adjust the model could help to make the case for keeping the cut where it should be (100%) while improving the overall experience of taking the assessment, particularly when questions are missed?
@Steve - spot on.
Bruce
...oh...and by the way...that's the sort of process that IDs are meant to help people through...
Not saying we always succeed for a variety of reasons, but that IS our job.
It would be nicer to build a course that is shortened if they already know what we are trying to tell you.
I envisage a course where you are asked questions throughout and if you get them right you almost do the speed readers version, if you get them wrong you are led through the process.
The assessment doesn't score you but instead chooses the content you need to view.
That's a good point, Phil. Elearning courses should probably never stand by themselves. Lots of compliance courses end up as a legal checkmark. It's convenience, I suppose:) But a training solution should want to change behavior, not cover an organization legally. Legal bulwark is something else entirely.
I've proposed that we have a series of legal acknowledgements for folks to read and indicate agreement as a foundation for values / legal issues. Rolling these things up in a training package masks the intent.
All these old habits we have to break and change:)
Love the concept of a tailoring assessment or a conversational component that helps the solution adapt to the needs / situation of the participant. Instead of the other way around:)
But Steve...that would cost money/time to do well! Perish the thought.
Maybe a little. But there are degrees of effort:) Take this model for example:
Shift it to this, where the assessment is graded by topic and the Topic Menu indicates completion by topic after the assessment is completed. The sections folks need to go through are clearly marked (tailored) by the assessment. Someone who already knows the info doesn't need to take the course at all. Someone who knows half, gets to take half. A little bit of additional effort to make this happen but it's largely a re-order. The side-benefit of providing those who already know the information an easy way to complete the module.
I just built a course, where you are guided through using a software tool. There are 4 sections and in each section you are asked questions (where do you press to launch...?) if you get any question wrong you are guided to a coach (the learning) who takes you through the process. This way you pass the assessments you never see the learning, if you fail you a question you do the learning. This in effect asks for 100% pass rate or else you do the learning.
This whole course is has a wrapper that makes it look like a cartoon and also gives rewards as you progress through the course. The biggest reward however is if you get 100% on each section you do not need to view the learning and spend less time.
I am loving this conversation, a lot of good points have been made! Thanks so much Bruce, Phil, and Steve for heeding the call and participating, love to hear your thoughts and opinions on this matter.
Thinking of my own past educational experiences (university, college, some online training, etc), interestingly, I never had to meet a passing score of 100%. Some very of good points for the 100% pass score! I suppose that, in the end, even if the learner gets 100% it doesn't "prove" much since we never know which answers were a lucky guess, and since quizzing typically only covers a sample of the material, not all.
The great thing here Louise is there is NO right or wrong answer to "what should a passing score be?". We all have to make decicions, as instructional designers, based on the variables and constraints of our own projects, so there's no one right answer. As pointed out in that handy reference link, at the end of the day, the passing score is a judgement call/opinion and as long as you can articulate solid reasoning for the score you've selected, based on your situation, which your boss/stakeholder agrees with, then I think you're headed in the right direction!
In your current situation, you might not have the option to re-design the courses. And if feedback indicates that, as they are, learners think the 100% pass is disheartening, it might make the most sense for now to lower the passing score to 80% and make it consistent across the board. Might be a good idea to schedule a meeting with management to discuss some of the business points brought up by Bruce, Phil, and Steve so that you can design your future courses in a way that best lets you incorporate a 100% pass rate, if you decide to go that way! :) Thanks again for bringing up this great topic, Louise!
I totally agree with you Nicole! There really is no "right" answer here, it is just a judgement call.
I like what David Glow said in this discussion about changing the culture of assessment so that scores become irrelevant and focus on providing feedback that actually helps the learner to better understand the concepts. However, I understand that in the context of corporate training, especially compliance training, that is not always possible.
I see what Bruce and Phil are saying about what setting the passing score 80% implies (that 20% of the content is not important) but at the same time, Nicole is right: in school we never had to get 100% to pass! One of the reasons getting a score of 100% may be "disheartening" to some is because the questions may be confusing or have several plausible answers.
For these reasons, I'm going to have to side with Nicole on this. It seems like in your case it would make sense to lower the score of existing courses to 80% and then think about possibly changing the way you build courses from here on out in order to try and take into account some of the ideas presented here.
Not sure if I ever came down agreeing on 100%. I would argue that an assessment in courses that are looking at behavioural changes of employees an assessment is a poor way of measuring this change and you should ideally look at your business metrics to see if there is a better way of measuring this.
I would recommend designing your courses differently so the user gets 100% but never actually knew they took an assessment, you would tailor the learning based on how they answer the questions and ensure that they get it correct before they are allowed to finish the course.
I really do not like the idea of assessments just for compliance sake and would always recommend doing it another way.
"Quizzes" are supposed to be a formative assessment (practice stuff). What I like to do is intersperse the "quiz" type of questions while showing how they help our employees improve their performance on the job. The summative assessments are questions on how the new learning is applied to their job role... so 100% is the norm, if anything is "missed" then feedback gives corrective action and the applied scenario is given again but in a different example. I would agree that if it's ok to miss 20% of the questions, let's take out 20% of the content to reduce cognitive overload and get 100% of whatever is left :o)
I find the hard part is for everyone to agree on what is need-to-know (even in a compliance course), but once we have that figured out, I take out the nice-to-knows and focus on what is necessary to improve job performance. I do not add anything to courses that is fluff or it'd be be neat if they knew this.
It is pretty easy to do this, I ask: give me an example of when this bit of information is USED during their normal daily tasks, if they can't, it's out. If they can, then we focus on 100% learning.
What we have to remember is that "learning" is when we change people's behavior. If they can't use the topics/concepts you are showing them, it's just nice to know, then there is no learning. But if there is learning to be done, we strive for 100%. It's ID mindset, it's management mindset, it's a work culture mind set.
But it works :o)
I am really enjoying the discussion in this post. My question is somewhat related which is how much push back is too much? At some point, whether it's good ID theory or not, when is it acceptable to give the customer what they want? The only reason I ask, is that I was involved in building compliance courses and the customers wanted everything in the "what not to do" category, (locked slides, text and audio, no scenarios or interactvitiy, Quiz with 80%) just forced "legal" learning. And, as much as I pushed, I finally felt I had to give in and just do what "they" wanted. Would love to hear your thoughts? Or maybe this is a different post for a different day. :)
That's always the rub. Folks have expectations and, largely, those expectations are that content is designed for their preferences and what they're used to and not what's good for the business requirements or for the participant.
Like any change, it really takes time. And folks need to see something new actually work before they buy into it. It's tough. I end up pushing back a bit on just about every project. I win a little, I lose a little. I usually give quite a bit in both compromise as well as going above and beyond expectations to show how things could be a little better. Making stuff suck a just little bit less over time. The conversation around business requirements and respect for the individuals participating in the experience is what matters. It's never a fight in a conflict sense, but selling something different is hard work. Like all selling:) First they have to know they need it.
It'll work over time. Expectations will shift little by little until (someday far into the future) we're all chuckling over how bad things were 20 years ago. The long game:)
Thanks for the input Steve. Maybe I need to start learning some Sales Techniques for Instructional Designers (there is a good book title).
A great title, although I have a preference for:
"Making stuff suck just little bit less over time." - Lose the battle, win the war. A handbook for instructional designers.
:)
(Copyright Perfect Performance Training 2015. All rights reserved. Based on an original idea by Steve Flowers)
This discussion is closed. You can start a new discussion or contact Articulate Support.