We currently have both multiple choice and true/false questions in our tests. Users must pass with 80% and can retake the test as many times as they need. I'm wondering what research shows about offering feedback for test questions. Should I be giving the correct answer in the feedback if the user guesses incorrectly?
Not sure whether it is "best practice" or not - but I always tell people that I can reduce their course development costs.
They ask how, and I explain we just need to strip out, and not bother with the 20% of the course build that people are allowed to get wrong in the Quiz.
Why 80%?
Why not 100%?
Most of the time it's that people do not want to annoy or upset their users, which (for me anyway), is a poor excuse.
Anyway - just a thought. I'm not sure of the "best practice" answer - but have some thoughts that may share when others have had a go.
I think it really depends on the purpose of the test.If you want to use the test as a training tool, it may be a good idea to provide customized feedback for each wrong answer explaining *why* it’s wrong (as opposed to just “incorrect”).And in that case I would usually have them then try again on that question, and they can’t move to the next question until the answer the current question correctly.Of course, that ensures that everyone will get 100%, because finishing the test means that you’ve answered everything correctly.
Obviously that would not be a good approach if what you’re aiming for is an actual assessment of who understands and who doesn’t!In that case I would allow one try per question, which sounds like what you’re doing.But if you’re going to let them take the test again, I wouldn’t tell them what the right answer is, because then the second attempt isn’t really a true measure of understanding.
In the simplest of cases I always try and ensure that:
1. If someone gets the question wrong, they understand what choosing that (incorrect) option will mean - what is caused by them choosing that action/task/option etc.
2. If (when) they get it right - reinforce the learning that led them to that correct choice by reiteration, rephrasing, and also perhaps adding in a "memorable fact" about the facts behind the correct answer.
I get annoyed when I see people designing tests where the pass rate is 80%, 4 of the 5 answers are stupid (and one is twice as long as the others), students are allowed to retake the test, but also allowed to review the quiz first time around (and there's no question bank!). A large % of all quizzes I see have all of these at once :(
Quizzes are one of the areas where I really feel there's an entire culture that needs to be broken - there are so many people out there that actually have no idea why they are designing them, how to make people LEARN from them rather than just memorise, and WHAT to do when people fail. Do not be afraid of the consequences that come from learners failing an examination/test/quiz/knowledge check.
Bruce, this post has been in the back of my mind since I saw it a couple of weeks ago… testing is something I’m passionate about (not passionate as in “let’s test everyone!” – more like “if we’re going to test, let’s do it right”).I’m totally with you on the stupid distractors, poorly written distractors, etc.And you’re right, it’s amazing how many times a test is put in place without any thought about what will happen if (when) someone fails it – what does failure mean?What is the remediation plan?
But… I’m struggling with the idea that the pass threshold should always be 100%.Maybe it depends on the purpose of the test… if it’s a learning opportunity then I’d probably agree with that.But if it’s a true assessment – we want to see what they’ve learned and/or we want to confirm a certain level of competency/mastery… maybe in that case 100% isn’t always the goal. If someone misses one question out of 50, is that really a fail?Of course, then you’re faced with the somewhat arbitrary decision of where to place the cutoff.
Anyway, thanks for giving me food for thought.I can’t say I’m convinced one way or the other – I may fall back on my trusty belief that “it depends on the situation” – but you definitely gave me something to think about!
I am, (mainly) talking about the "...and now...10 questions to check you have understood the course....". Despite protestation from me, sometimes clients cannot be convinced otherwise. In those cases I like to try and insist on 100%.
If they only want 80%, I often ask which bits of the course we can cut out, (saving money....), because it's not essential that the learners understand them
The lack of planning for "What happens if they fail?" is so common it beggars belief sometimes.
If someone misses/fails 1 question in 50, and that was the question about the thing that cost the company a lawsuit, or someone died, (because the learner did not understand the principle correctly...), then yes, that is an EPIC fail.
If someone misses/fails 1 question in 50, and that was the question about the thing that cost the company a lawsuit, or someone died, (because the learner did not understand the principle correctly...), then yes, that is an EPIC fail.
I guess the way I think of it is, if you’re training on a principle that’s related to something really important, why would you ask only one question about it?I’d probably want to write several questions that attack the issue/topic from different directions, address different aspects of the topic/principles, etc.
Actually, now that I really think about it, if something were truly about life or death decisions or significant legal liability, I wouldn’t want to be addressing that with a multiple-choice type test anyway.I’d much prefer something performance based.If you’ve got a firefighter who scored 100% on a M/C test and one who scored 95% on a performance-based test, I’d rather have the guy with the 95% responding when my house is on fire.
Of course I have now moved way off topic.Sorry about that!
Great topic guys. I've really enjoyed reading through your positions. I too am often surprised by the assessment standards (or lack) that I see. More often than not they seem like afterthoughts that someone threw in because the course is supposed to have a test in it. Probably because that's exactly what happens. What are they even trying to measure with those? If it's meant to measure how well the user understood each section, they're missing the boat. More often than not all that is getting measured is how easy,hard, tricky, ridiculous the questions are written- all of which is irrelevant if they can keep taking it over. Even a well written assessment could just be a measurement of how good the user is at logic. Since I don't see reliability in the measurements, I'd just as soon use them for a different purpose.
I guess you could call me an assessment skeptic. I still like them, though. For the topics I work on, most of my measurement comes outside of the course. When I think about what I really want my assessments to do if not measuring, it's to reinforce the learning objectives and even continue the learning experience. That makes me a big fan of feedback. Whenever possible, the feedback includes what would happen if they chose that answer in real life
I'm not saying this is the best way or even the right way. It's just how I've come to use them in my situation.
Looking forward to seeing what else pops up in the convo! Thanks for posting, Jill.
First, there is another topic on good and bad multiple choice questions so I'll not add anything further on this.
Second, I think it depends on the purpose of the end test. If it is for the learner to gauge their learning, then I have no issue with a nominal pass mark, but if we take it back to basics:
we start with a set of learning objectives that have to be achieved
each of these need to be tested to ensure they have been achieved
failure to answer the question correctly means failure to achieve the objective
So, coming back to what Bruce states, 100% is the target every time.
Is 100% realistic? If you have 4 objectives, these may be adequately tested with 4 well crafted questions. But I suggest that in most cases more than one question is required to test the objective and this is where you might permit some incorrect answers - if you have 10 questions per objective allowing for some mistakes,may be acceptable. What you then need to do is give those key questions a weighting or value such that if they are answered incorrectly the learner cannot "pass".
The issue then is, can you come up with sufficient questions? Will the client pay for the design of this number of questions?
The answer is neither black nor white, but often grey is not available as an option!
I find that if you include feedback for incorrect answers in a post-test, you are in essence ingraining what the user didn't know. It's an "aha... I got this wrong and if I'm going to pass this test, this is what I need to know." It also offers an opportunity for them to think about it and rationalize why they missed it (or perhaps argue with me because they think they are right). I could have my users have to go back through the content to find the answer, but that then becomes a frustration to them and a loss of productivity for the client. More than likely, they'll just re-launch the test and try another answer, which doesn't provide the education they received by giving it to them in the incorrect feedback. And if I don't require a 100% score and they only miss 1 or 2, then they never get that feedback.
For compliance courses, I require 100%. For other courses, I typically require 80% with feedback given for incorrect answers. Our healthcare clients have to take a lot of annual training to do and the ultimate goal is that they understand the concepts; not spend all day retaking courses/tests.
OK - so maybe 100% is me just trying to make people think....
There is a point to it though - when clients say "80%", I always ask why?
I always ask which 20% of the learning do you want to scrub, 'coz it does not matter. It gets people thinking.
Perhaps they want to give an incentive for getting 80% - 100%?
Perhaps above 90% gets them invited to a "thought-leaders" session?
I don't know - but I DO know that people never really give it a moment of thought in many cases before saying "80% pass rate".
I would like to think (naively?) that people would not get frustrated when they get answers incorrect, but would regard it as a GOAL to find the answer - that the answer really mattered to them and their personal success.
Yes - I live in a world of pipe-dreams I know, however, perhaps it's a realistic goal to design for...
I certainly don't think wanting 100% is a pipe dream. There are situations when no less in acceptable. I don't want a surgeon who is learning the principles if brain surgery or a pilot who is learning to fly a jet with 200+ passengers at 35,000 feet to get anything less. However, I do believe that there are situations (e.g., customer service training, time management) when 80% may be fine; especially if you've designed solid, critical-thinking questions with good distractors and you use the opportunity to teach them when they've missed one (feedback). Consider an organization with 1000 employees, who requires each employee to take 10 online courses during the year, and each course requires 100%. The majority of employees would miss at least one question per course requiring a re-launch to re-take the test. Even if that was just 5 minutes, it could equal more than $10,000 in training dollars, lost productivity, etc. in the year.
It may be settling, but I don't think perfect is necessary every time.
15 Replies
Hi Jill.
Not sure whether it is "best practice" or not - but I always tell people that I can reduce their course development costs.
They ask how, and I explain we just need to strip out, and not bother with the 20% of the course build that people are allowed to get wrong in the Quiz.
Why 80%?
Why not 100%?
Most of the time it's that people do not want to annoy or upset their users, which (for me anyway), is a poor excuse.
Anyway - just a thought. I'm not sure of the "best practice" answer - but have some thoughts that may share when others have had a go.
Bruce
I think it really depends on the purpose of the test. If you want to use the test as a training tool, it may be a good idea to provide customized feedback for each wrong answer explaining *why* it’s wrong (as opposed to just “incorrect”). And in that case I would usually have them then try again on that question, and they can’t move to the next question until the answer the current question correctly. Of course, that ensures that everyone will get 100%, because finishing the test means that you’ve answered everything correctly.
Obviously that would not be a good approach if what you’re aiming for is an actual assessment of who understands and who doesn’t! In that case I would allow one try per question, which sounds like what you’re doing. But if you’re going to let them take the test again, I wouldn’t tell them what the right answer is, because then the second attempt isn’t really a true measure of understanding.
That’s my two cents, for whatever it’s worth!
In the simplest of cases I always try and ensure that:
1. If someone gets the question wrong, they understand what choosing that (incorrect) option will mean - what is caused by them choosing that action/task/option etc.
2. If (when) they get it right - reinforce the learning that led them to that correct choice by reiteration, rephrasing, and also perhaps adding in a "memorable fact" about the facts behind the correct answer.
I get annoyed when I see people designing tests where the pass rate is 80%, 4 of the 5 answers are stupid (and one is twice as long as the others), students are allowed to retake the test, but also allowed to review the quiz first time around (and there's no question bank!). A large % of all quizzes I see have all of these at once :(
Quizzes are one of the areas where I really feel there's an entire culture that needs to be broken - there are so many people out there that actually have no idea why they are designing them, how to make people LEARN from them rather than just memorise, and WHAT to do when people fail. Do not be afraid of the consequences that come from learners failing an examination/test/quiz/knowledge check.
Bruce
Bruce, this post has been in the back of my mind since I saw it a couple of weeks ago… testing is something I’m passionate about (not passionate as in “let’s test everyone!” – more like “if we’re going to test, let’s do it right”). I’m totally with you on the stupid distractors, poorly written distractors, etc. And you’re right, it’s amazing how many times a test is put in place without any thought about what will happen if (when) someone fails it – what does failure mean? What is the remediation plan?
But… I’m struggling with the idea that the pass threshold should always be 100%. Maybe it depends on the purpose of the test… if it’s a learning opportunity then I’d probably agree with that. But if it’s a true assessment – we want to see what they’ve learned and/or we want to confirm a certain level of competency/mastery… maybe in that case 100% isn’t always the goal. If someone misses one question out of 50, is that really a fail? Of course, then you’re faced with the somewhat arbitrary decision of where to place the cutoff.
Anyway, thanks for giving me food for thought. I can’t say I’m convinced one way or the other – I may fall back on my trusty belief that “it depends on the situation” – but you definitely gave me something to think about!
Sheila,
You are right, it depends on the situation.
I am, (mainly) talking about the "...and now...10 questions to check you have understood the course....". Despite protestation from me, sometimes clients cannot be convinced otherwise. In those cases I like to try and insist on 100%.
If they only want 80%, I often ask which bits of the course we can cut out, (saving money....), because it's not essential that the learners understand them
The lack of planning for "What happens if they fail?" is so common it beggars belief sometimes.
If someone misses/fails 1 question in 50, and that was the question about the thing that cost the company a lawsuit, or someone died, (because the learner did not understand the principle correctly...), then yes, that is an EPIC fail.
Bruce
I guess the way I think of it is, if you’re training on a principle that’s related to something really important, why would you ask only one question about it? I’d probably want to write several questions that attack the issue/topic from different directions, address different aspects of the topic/principles, etc.
Actually, now that I really think about it, if something were truly about life or death decisions or significant legal liability, I wouldn’t want to be addressing that with a multiple-choice type test anyway. I’d much prefer something performance based. If you’ve got a firefighter who scored 100% on a M/C test and one who scored 95% on a performance-based test, I’d rather have the guy with the 95% responding when my house is on fire.
Of course I have now moved way off topic. Sorry about that!
Great topic guys. I've really enjoyed reading through your positions. I too am often surprised by the assessment standards (or lack) that I see. More often than not they seem like afterthoughts that someone threw in because the course is supposed to have a test in it. Probably because that's exactly what happens. What are they even trying to measure with those? If it's meant to measure how well the user understood each section, they're missing the boat. More often than not all that is getting measured is how easy,hard, tricky, ridiculous the questions are written- all of which is irrelevant if they can keep taking it over. Even a well written assessment could just be a measurement of how good the user is at logic. Since I don't see reliability in the measurements, I'd just as soon use them for a different purpose.
I guess you could call me an assessment skeptic. I still like them, though. For the topics I work on, most of my measurement comes outside of the course. When I think about what I really want my assessments to do if not measuring, it's to reinforce the learning objectives and even continue the learning experience. That makes me a big fan of feedback. Whenever possible, the feedback includes what would happen if they chose that answer in real life
I'm not saying this is the best way or even the right way. It's just how I've come to use them in my situation.
Looking forward to seeing what else pops up in the convo! Thanks for posting, Jill.
First, there is another topic on good and bad multiple choice questions so I'll not add anything further on this.
Second, I think it depends on the purpose of the end test. If it is for the learner to gauge their learning, then I have no issue with a nominal pass mark, but if we take it back to basics:
So, coming back to what Bruce states, 100% is the target every time.
Is 100% realistic? If you have 4 objectives, these may be adequately tested with 4 well crafted questions. But I suggest that in most cases more than one question is required to test the objective and this is where you might permit some incorrect answers - if you have 10 questions per objective allowing for some mistakes,may be acceptable. What you then need to do is give those key questions a weighting or value such that if they are answered incorrectly the learner cannot "pass".
The issue then is, can you come up with sufficient questions? Will the client pay for the design of this number of questions?
The answer is neither black nor white, but often grey is not available as an option!
Weighted questions... interesting. I've never even thought about that. Anyone know if that's an option in Storyline?
Yep indeedy....
Set up in Question Banks...
Bruce
Certainly you can. The default point value is 10. You can reduce non crucial to 1 point and even increase the key questions to 70 or 80 or even 1000!
I really wanted to assume it did, but but I've learned that lesson already. Thanks guys
I find that if you include feedback for incorrect answers in a post-test, you are in essence ingraining what the user didn't know. It's an "aha... I got this wrong and if I'm going to pass this test, this is what I need to know." It also offers an opportunity for them to think about it and rationalize why they missed it (or perhaps argue with me because they think they are right). I could have my users have to go back through the content to find the answer, but that then becomes a frustration to them and a loss of productivity for the client. More than likely, they'll just re-launch the test and try another answer, which doesn't provide the education they received by giving it to them in the incorrect feedback. And if I don't require a 100% score and they only miss 1 or 2, then they never get that feedback.
For compliance courses, I require 100%. For other courses, I typically require 80% with feedback given for incorrect answers. Our healthcare clients have to take a lot of annual training to do and the ultimate goal is that they understand the concepts; not spend all day retaking courses/tests.
OK - so maybe 100% is me just trying to make people think....
There is a point to it though - when clients say "80%", I always ask why?
I always ask which 20% of the learning do you want to scrub, 'coz it does not matter. It gets people thinking.
Perhaps they want to give an incentive for getting 80% - 100%?
Perhaps above 90% gets them invited to a "thought-leaders" session?
I don't know - but I DO know that people never really give it a moment of thought in many cases before saying "80% pass rate".
I would like to think (naively?) that people would not get frustrated when they get answers incorrect, but would regard it as a GOAL to find the answer - that the answer really mattered to them and their personal success.
Yes - I live in a world of pipe-dreams I know, however, perhaps it's a realistic goal to design for...
Bruce
I certainly don't think wanting 100% is a pipe dream. There are situations when no less in acceptable. I don't want a surgeon who is learning the principles if brain surgery or a pilot who is learning to fly a jet with 200+ passengers at 35,000 feet to get anything less. However, I do believe that there are situations (e.g., customer service training, time management) when 80% may be fine; especially if you've designed solid, critical-thinking questions with good distractors and you use the opportunity to teach them when they've missed one (feedback). Consider an organization with 1000 employees, who requires each employee to take 10 online courses during the year, and each course requires 100%. The majority of employees would miss at least one question per course requiring a re-launch to re-take the test. Even if that was just 5 minutes, it could equal more than $10,000 in training dollars, lost productivity, etc. in the year.
It may be settling, but I don't think perfect is necessary every time.
This discussion is closed. You can start a new discussion or contact Articulate Support.