I'm currently tasked with producing an evaluation form for learners in our pilot group to complete. I'm not entirely certain what questions should be included and was wondering if you had any suggestions.
As you might have guessed the purpose of the evaluation form is to capture the user experience and address any issues before the module goes live.
On a personal note in the past when I have done pilots the things I have really focused on are 1) navigation (where they got hung up, didn't know where to click, what to do, etc.) and 2) content (did they understand it? does it relate to their job? is it meaningful?). I provided ways for the testers to document their findings or feedback. I also once was able to actually record my testers (with a video camera and a screen recording tool) as they completed the course, which gave me additional insights into where they got stuck, how long it took to do certain parts, where they hesitated or clicked in the wrong places, etc. etc. I got a lot of good information from doing this. And then I adjusted my course accordingly. Good luck!
Thank you very much for all the responses, I'll bear them in mind when designing my evaluation form.
Also...
Do you ask your pilot group to fill in the evaluation reflecting on the whole module or section by section?
Thanks
Harri
I think this depends on the length of the module to be honest. If it is a long package or course then I would split it down to manageable chunks. At the end of the day you want feedback that is constructive so asking them to try and remember 60 minutes of a course at once may be a struggle.
One of the things I find helpful is making sure your pilot group has access to the evaluation questions in advance, so they are aware of what they should be looking for. I know I've gotten surveys after the fact for things and wish I could go back and double-check something. We used Survey Monkey and sent out the evaluation link with the invite to pilot.
Also, depending on the complexity and scope of what you're doing, you may try pilot groups with different focuses. We had a broad, global group pilot a course for a wide variety of things (content, navigation, design, etc.) and then a couple of smaller groups let us know about more specific things (how was the pace and narration as someone who speaks English as a second language? as a newer employee, how would you rate your understanding of the content before and after?)
Thank you for your suggestions - I will be sending the evaluation form out in advance to the pilot group leaders but it's probably a good idea for me to ensure they pass it on to the participants in advance too.
Harri, For pilots, I like to use what I call "confidence surveys." I base the questions off the learning objectives of the course. For example, if the course topic is How to Change a Tire, I ask learners to rate their confidence (on a scale of 1-5) in their ability to use a jack, tighten lug nuts, etc. I provide learners the same confidence survey before they take the pilot course AND after they complete the course. This way you can measure the learning gain from having completed the course. For example, if a learner rated their confidence with log nuts as a "1" and after the course as a "5" then you can attribute the gain directly to having taken the course. Good luck!
That's great idea, especially for modules which are trying to teach a skill - unfortunately my module's more about rolling out company policy so I'm not sure how valuable that type of evaluation form would be in this instance. But it's definitely a thought for future modules.
11 Replies
I'd want to know the usual things like:
I'd then want to know some usability information such as:
A few I've used in the past:
What is your overall impression of the quality of this course?
Have you taken other courses on this subject? (Yes/No)
How did this course compare?
What did you like about the course?
What could be improved?
Would you recommend this course to a colleague?
Give a free text box that asks them for any changes they would make. I always find that free evaluation provides some of the most helpful answers.
Hi Harri,
Here's a few past forum discussions about e-learning pilots that might be useful:
On a personal note in the past when I have done pilots the things I have really focused on are 1) navigation (where they got hung up, didn't know where to click, what to do, etc.) and 2) content (did they understand it? does it relate to their job? is it meaningful?). I provided ways for the testers to document their findings or feedback. I also once was able to actually record my testers (with a video camera and a screen recording tool) as they completed the course, which gave me additional insights into where they got stuck, how long it took to do certain parts, where they hesitated or clicked in the wrong places, etc. etc. I got a lot of good information from doing this. And then I adjusted my course accordingly. Good luck!
Hi Everyone,
Thank you very much for all the responses, I'll bear them in mind when designing my evaluation form.
Also...
Do you ask your pilot group to fill in the evaluation reflecting on the whole module or section by section?
Thanks
Harri
I think this depends on the length of the module to be honest. If it is a long package or course then I would split it down to manageable chunks. At the end of the day you want feedback that is constructive so asking them to try and remember 60 minutes of a course at once may be a struggle.
One of the things I find helpful is making sure your pilot group has access to the evaluation questions in advance, so they are aware of what they should be looking for. I know I've gotten surveys after the fact for things and wish I could go back and double-check something. We used Survey Monkey and sent out the evaluation link with the invite to pilot.
Also, depending on the complexity and scope of what you're doing, you may try pilot groups with different focuses. We had a broad, global group pilot a course for a wide variety of things (content, navigation, design, etc.) and then a couple of smaller groups let us know about more specific things (how was the pace and narration as someone who speaks English as a second language? as a newer employee, how would you rate your understanding of the content before and after?)
Hi Laura,
Thank you for your suggestions - I will be sending the evaluation form out in advance to the pilot group leaders but it's probably a good idea for me to ensure they pass it on to the participants in advance too.
Harri, For pilots, I like to use what I call "confidence surveys." I base the questions off the learning objectives of the course. For example, if the course topic is How to Change a Tire, I ask learners to rate their confidence (on a scale of 1-5) in their ability to use a jack, tighten lug nuts, etc. I provide learners the same confidence survey before they take the pilot course AND after they complete the course. This way you can measure the learning gain from having completed the course. For example, if a learner rated their confidence with log nuts as a "1" and after the course as a "5" then you can attribute the gain directly to having taken the course. Good luck!
I agree with the post above to leave an open box for open-ended responses. I believe you want your feedback to be as thorough as possible.
Hi Kimberly,
That's great idea, especially for modules which are trying to teach a skill - unfortunately my module's more about rolling out company policy so I'm not sure how valuable that type of evaluation form would be in this instance. But it's definitely a thought for future modules.
This discussion is closed. You can start a new discussion or contact Articulate Support.