I'm new to this whole e-learning game, and I'm just wondering whether or not you think quizzing is really an effective way of testing the learners' knowledge. I don't want to quiz just to quiz.
Welcome to the community! That is a really great question, and I'm afraid I don't have a black and white answer for you. I would say that it really depends on how you go about it.
While having a quiz is a requirement for a lot of courses, I think you should avoid including a quiz "just because" whenever possible. If you decide to include a quiz, it should be there to help you (and the learner!) gauge how well they understood the content. So how do you go about doing that? Come up with questions that make learner's think and/or apply their knowledge, not just recognize or recall.
As a minimum I normally want to try and get companies to do a "quiz" at least a week after completion of the course. Quizzes at the end of a course normally just check the learner has a working short-term memory, but I am much more in favour of some sort of "on the job" test that learning has taken place and skills have been transferred.
Saying that - many client want a quiz because that is what they expect.
First tip- stop saving quizzing until the end. Test frequently along the way, check for understanding and correct things before misinformation sets in. Ideally, have them practice skills and give coaching (hey- there's a sports idea that we don't apply well). Don't save checking skills for game day and see a bad situation play out on the scoreboard.
Another item- change the culture of assessment. That is- why are we grading?- i.e. "you must pass this with a score of 80%". Yes, in some situations this applies, in many, it really doesn't.
I often design assessments to be a test applying skills to a business process (you know- work!).
Perform step 1, 2, 3... and all along the way, you get feedback. Stumble on step 2? We tell you so. We stop you, give you feedback, correct you, have you re-perform, and once you do it right then you are ready to move on. No scores- just coaching and feedback. (Feedback, not spoonfeeding correct answers).
At the end of the course, everyone "gets 100%" (because they completed/crossed the finish line). But behind the scenes, we analyze the data. What steps took the most attempts? What scenarios were most challenging, etc...
This is NOT done to hold the employee accountable. It is to hold L&D accountable. We need to see these issues and figure out ways to better support performance. We move to the issues.
It isn't "Employee, you only scored 80% and you need 90% to 'pass'".
It is "Employee, we'll stick with you until you get there. We'll look at the sticking points and do something to help".
The score (data) is for us, not them, frankly.
Again- there ARE exceptions (I work in environments with a lot of regulatory oversight and sometimes they demand the other designs). But for the most part, it is about providing them information and practice, with lots of feedback loops. Letting them try until it clicks for them.
@Bruce: Having the learners take the quiz a week after they complete the course is a great idea. In my experience, I have never seen this technique implemented. Do you find that the results are more telling? What do you do if the person does not pass? Do they have to retake the course?
@David: I totally agree with you on this one. Quizzes should be part of the learning experience, and not simply a way to assign a score/grade. Unfortunately, like you pointed out, sometimes a score/grade is necessary for regulatory/compliance reasons.
Not sure they are more "telling", but certainly check to see if they learned anything. I still prefer "real life" quizzes. I once remember a general eLearning strategy coming down from a certain Mr L. Ellison and his team that said (in so many words...) "We are not going to have end of course tests, we'll know if they learn from the course because they will be able to go out and install the software".
As to what happens, that's the biggest failing in the entire eLearning "Quiz" process, in many cases there is no incentive to get it right. We allow people to review (so they know the answers...), we pass at 80%, (which 20% of the course do you want me to cut out, saving me time, and you money?), or we do nothing.
Not sure retaking the course is always very motivational, but I do have clients that want that, but there should be SOMETHING happening. Very often people cannot even be bothered to report on/action "fails" from LMS score reports.
Maybe I come from a slightly harsher business background than many, but "Quizzes", and the way they are administered, in the vast majority of cases that I have seen, are a complete waste of time.
Thanks everyone for your responses. I agree that making quizzes more about learning and less about evaluation is the way to go. I will try to keep that in mind while developing my next courses.
@Bruce: I completely agree with you. I believe that if people understand the benefit of the course in terms of practical application to their job, they will likely pay attention and try to get something out of it. And like you said, if they don't it will be obvious based on their ability/inability to apply what they've learned to their job.
I don't necessarily think retaking the course is very motivational either, but based on past experience I can image clients wanting them to do that if they don't pass.
Just to add my 2 cents, I think that your quizzes should always be performance based. Instead of quizzing for factual recall, test to see whether or not the user can actually complete form A203 without immediate feedback or test whether or not they can successfully make a sale in a given scenario by trying to convince a customer you've created. That's where quizzing has the added value.
Just to add my 2 cents, I think that your quizzes should always be performance based. Instead of quizzing for factual recall, test to see whether or not the user can actually complete form A203 without immediate feedback or test whether or not they can successfully make a sale in a given scenario by trying to convince a customer you've created. That's where quizzing has the added value.
It also takes about 20 times longer to plan, produce and Pilot. That's where a lot of organisations give up and go for the "20 randomised multiple choice" option :)
For sure, it can take longer but I don't believe that it has to, especially if you go into the kickoff and planning phase with that kind of quizzing in mind. I think that the other thing to keep in mind is that a scenario really is just a series of multiple choice questions. If you create a relevant and engaging story, a la interactive fiction and text adventure games, you can get away with creating that scenario text-only.
Ask key stakeholders: What do we need folks to DO? How do we know they can DO it? (Shameless endorsement of a great resource: lead them through Cathy Moore's Action Mapping).
Gets the conversation to "what do we really need here?" vs "what questions can we write?".
You won't always win- there are always some folks who feel some recall questions *must* be asked, but I've used the strategy with a team VERY deeply embedded in "bore and score" designs, and broke ground in most cases.
Ask key stakeholders: What do we need folks to DO? How do we know they can DO it? (Shameless endorsement of a great resource: lead them through Cathy Moore's Action Mapping).
Gets the conversation to "what do we really need here?" vs "what questions can we write?".
You won't always win- there are always some folks who feel some recall questions *must* be asked, but I've used the strategy with a team VERY deeply embedded in "bore and score" designs, and broke ground in most cases.
I seem to be unable to find the "Every ID should read this and use it to guide their every working breath" button.....
Ask key stakeholders: What do we need folks to DO? How do we know they can DO it? (Shameless endorsement of a great resource: lead them through Cathy Moore's Action Mapping).
Gets the conversation to "what do we really need here?" vs "what questions can we write?".
Hey guys! Just thought I'd pop back in here and let you know that I posted an article inspired by this discussion. Check it out and let me know what you think.
17 Replies
Hey Brian!
Welcome to the community! That is a really great question, and I'm afraid I don't have a black and white answer for you. I would say that it really depends on how you go about it.
While having a quiz is a requirement for a lot of courses, I think you should avoid including a quiz "just because" whenever possible. If you decide to include a quiz, it should be there to help you (and the learner!) gauge how well they understood the content. So how do you go about doing that? Come up with questions that make learner's think and/or apply their knowledge, not just recognize or recall.
Anyone else want to weigh in on this?
For! But with specific conditions
http://businesscriticallearning.com/assess-more-to-coach-not-judge/
As a minimum I normally want to try and get companies to do a "quiz" at least a week after completion of the course. Quizzes at the end of a course normally just check the learner has a working short-term memory, but I am much more in favour of some sort of "on the job" test that learning has taken place and skills have been transferred.
Saying that - many client want a quiz because that is what they expect.
First tip- stop saving quizzing until the end. Test frequently along the way, check for understanding and correct things before misinformation sets in. Ideally, have them practice skills and give coaching (hey- there's a sports idea that we don't apply well). Don't save checking skills for game day and see a bad situation play out on the scoreboard.
Another item- change the culture of assessment. That is- why are we grading?- i.e. "you must pass this with a score of 80%". Yes, in some situations this applies, in many, it really doesn't.
I often design assessments to be a test applying skills to a business process (you know- work!).
Perform step 1, 2, 3... and all along the way, you get feedback. Stumble on step 2? We tell you so. We stop you, give you feedback, correct you, have you re-perform, and once you do it right then you are ready to move on. No scores- just coaching and feedback. (Feedback, not spoonfeeding correct answers).
At the end of the course, everyone "gets 100%" (because they completed/crossed the finish line). But behind the scenes, we analyze the data. What steps took the most attempts? What scenarios were most challenging, etc...
This is NOT done to hold the employee accountable. It is to hold L&D accountable. We need to see these issues and figure out ways to better support performance. We move to the issues.
It isn't "Employee, you only scored 80% and you need 90% to 'pass'".
It is "Employee, we'll stick with you until you get there. We'll look at the sticking points and do something to help".
The score (data) is for us, not them, frankly.
Again- there ARE exceptions (I work in environments with a lot of regulatory oversight and sometimes they demand the other designs). But for the most part, it is about providing them information and practice, with lots of feedback loops. Letting them try until it clicks for them.
@Bruce: Having the learners take the quiz a week after they complete the course is a great idea. In my experience, I have never seen this technique implemented. Do you find that the results are more telling? What do you do if the person does not pass? Do they have to retake the course?
@David: I totally agree with you on this one. Quizzes should be part of the learning experience, and not simply a way to assign a score/grade. Unfortunately, like you pointed out, sometimes a score/grade is necessary for regulatory/compliance reasons.
Not sure they are more "telling", but certainly check to see if they learned anything. I still prefer "real life" quizzes. I once remember a general eLearning strategy coming down from a certain Mr L. Ellison and his team that said (in so many words...) "We are not going to have end of course tests, we'll know if they learn from the course because they will be able to go out and install the software".
As to what happens, that's the biggest failing in the entire eLearning "Quiz" process, in many cases there is no incentive to get it right. We allow people to review (so they know the answers...), we pass at 80%, (which 20% of the course do you want me to cut out, saving me time, and you money?), or we do nothing.
Not sure retaking the course is always very motivational, but I do have clients that want that, but there should be SOMETHING happening. Very often people cannot even be bothered to report on/action "fails" from LMS score reports.
Maybe I come from a slightly harsher business background than many, but "Quizzes", and the way they are administered, in the vast majority of cases that I have seen, are a complete waste of time.
Loving @David's views and approach.
Thanks everyone for your responses. I agree that making quizzes more about learning and less about evaluation is the way to go. I will try to keep that in mind while developing my next courses.
@Bruce: I completely agree with you. I believe that if people understand the benefit of the course in terms of practical application to their job, they will likely pay attention and try to get something out of it. And like you said, if they don't it will be obvious based on their ability/inability to apply what they've learned to their job.
I don't necessarily think retaking the course is very motivational either, but based on past experience I can image clients wanting them to do that if they don't pass.
Just to add my 2 cents, I think that your quizzes should always be performance based. Instead of quizzing for factual recall, test to see whether or not the user can actually complete form A203 without immediate feedback or test whether or not they can successfully make a sale in a given scenario by trying to convince a customer you've created. That's where quizzing has the added value.
I totally agree, Kristin. Very well said!
It also takes about 20 times longer to plan, produce and Pilot. That's where a lot of organisations give up and go for the "20 randomised multiple choice" option :)
For sure, it can take longer but I don't believe that it has to, especially if you go into the kickoff and planning phase with that kind of quizzing in mind. I think that the other thing to keep in mind is that a scenario really is just a series of multiple choice questions. If you create a relevant and engaging story, a la interactive fiction and text adventure games, you can get away with creating that scenario text-only.
Easiest way to break the mold?
Ask key stakeholders: What do we need folks to DO? How do we know they can DO it?
(Shameless endorsement of a great resource: lead them through Cathy Moore's Action Mapping).
Gets the conversation to "what do we really need here?" vs "what questions can we write?".
You won't always win- there are always some folks who feel some recall questions *must* be asked, but I've used the strategy with a team VERY deeply embedded in "bore and score" designs, and broke ground in most cases.
I seem to be unable to find the "Every ID should read this and use it to guide their every working breath" button.....
Well said! Thanks for sharing :)
Great tips, Kristin! Our minds are in sync when it comes to this...
Hey guys! Just thought I'd pop back in here and let you know that I posted an article inspired by this discussion. Check it out and let me know what you think.
This discussion is closed. You can start a new discussion or contact Articulate Support.