I am developing my very first e-learning course using Storyline and have included testing in my timeline. Do you give your testers a questionnaire or any guidance for the type of feedback you'd like or do you just have them take the course and let you know if anything is broken? I'd appreciate getting substantive feedback from them so I can improve the course and my skills. How can I ensure that I get this?
If you have any samples or templates, I would greatly appreciate them. Thanks!
I'd say less information is good. Have them note any technical issues that may arise and anywhere they were confused (maybe how to navigate or by the content--e.g.,"this question doesn't make sense")
Perhaps someone in the community has a beta tester checklist they could share?
Areas where learning is strong/useful and/or weak/redundant.
Technical/useability issues.
I try not to be too prescriptive with a "form", and let the feedback come freeform. May be harder for me to interpret, but seems to elicit more feedback, which is what the exercise is about
If you can combine the ability to watch your users and observe how they interact with your course, you'll see the areas of confusing navigation. If you couple this with some feedback channel, you'll get broader input. If you can't actually watch them, think of ways that you could do this - maybe a remote screenshare or a service like usertesting.com. You could also have them do a screencast describing what they are thinking rather than writing down. You might get more candid feedback.
I like to send out smaller tests to folks during the formative changes to see if the navigation makes sense, instructions are clear, messages are consistent, etc.. I love the idea of asking folks to do a screencast if they have the tools and are comfortable doing that.
When I test with larger groups, I want to see if the strategy is moving the needle. We have a testing package we use to measure before and after snapshots of abilities and perceptions. It's pretty low tech. All paper based.
The pre-module survey and assessment contents vary based on the context. Generally we're looking for the participant to describe how they feel about a particular topic before hand, how much they know, what their understanding of rules and consequences are, etc.. This is brief as we don't want to over-influence the experience with prep, but we do want to get some sense of where folks are before. We also assess various tasks on paper ahead of time and extend that capture with a confidence measurement (How confident are you in that answer?)
We offer a similar assessment after the module experience that measures similar task contexts and asks the same confidence questions. This gives us a general idea whether the module did what it was designed to do. These tests are NOT part of the final deployment, though we do try to put L3 measures (are you using this on the job and can you accomplish the task at some point in the future) to evaluate how long the results last. If the L3 measures are off the mark and don't jive with our user test results, we make an adjustment or add something a little different to span the gap. It's all you can do, really
These are awesome recommendations! I like the "watching" the learners idea too. I did that for usability testing one time and it was a big eye-opener. I'll combine that with some form of open-ended feedback option.
Should have mentioned this after reading Daniel's suggestion. We put together a set of Acceptance Testing Checklists. You may be able to extract a few things from this list of factors.
I think it really depends on what you want from your beta testers. In my experience, if you are using non-eLearning people, they're unlikley to point out any major/minor design/technical issues unless you ask them VERY specific questions. They just don't know any better. In most cases, the small details we'd like them to point out, they won't even notice.
Typically, if I'm having my SME(s) look at the content, I'll ask them to ignor the "look and feel" of the course and simply focus on content accuracy. Is everything accurate? Is the course conveying the complete message that you want? Etc.
If I'm having a group of end-users from the targeted audience look at the course, I'll simply ask them to give general feedback on the course. With this group, you'll likely have to ask a lot of questions to get the right info from them. For example, was their anything you liked/disliked about how the course looked? Do you feel the content helped you to meet the specified learning objectives? Etc. Again, unless they've reviewed eLearning many times in the past, they won't know to give you this type of feedback.
And lastly, if I'm looking for feedback on how the course looks, it's structure and/or design; I'll have a group of my peers look at the course. They'll be the ones who really understand what you're looking for in terms of feedback AND they are the best group for this type of review.
I recently beta tested a course I developed by having two people complete the course while I watched. This works for me because I am in a corporate setting. I was able to observe navigation issues as well as measure engagement. I did not interact with the participant while they completed the course but did interview them afterward. Of couse I had several others complete the course without me and then I interviewed each one. I highly recommend observing if you can.
I think observing users take the course can have a ton of benefits. Never thought of that. Do you preface anything for them before they start (i.e., act like I'm not here, interact with the course as you normally would, etc.)?
In a previous project, I asked testers to narrate their activities. The narrative helped cover the areas when the user seemed confused but we couldn't tell why.
Not all testers were comfortable narrating, and some were better at it than others.
Thanks for the great ideas. I'm getting closer to testing and feel better knowing that I have some best practices to use. I made my first course far too involved, so hopefully, there won't be a ton of technical things to fix.
Love this community! Thanks for the great ideas and feedback. This gives me a lot of confidence to do something I have not done before and be really confident about my work.
15 Replies
Michelle:
I'd say less information is good. Have them note any technical issues that may arise and anywhere they were confused (maybe how to navigate or by the content--e.g.,"this question doesn't make sense")
Perhaps someone in the community has a beta tester checklist they could share?
Generally, (in my experience...), the areas are:
I try not to be too prescriptive with a "form", and let the feedback come freeform. May be harder for me to interpret, but seems to elicit more feedback, which is what the exercise is about
Bruce
Michelle -
Great step to get feedback from your learners.
If you can combine the ability to watch your users and observe how they interact with your course, you'll see the areas of confusing navigation. If you couple this with some feedback channel, you'll get broader input. If you can't actually watch them, think of ways that you could do this - maybe a remote screenshare or a service like usertesting.com. You could also have them do a screencast describing what they are thinking rather than writing down. You might get more candid feedback.
I hope that helps.
Holly
I like to send out smaller tests to folks during the formative changes to see if the navigation makes sense, instructions are clear, messages are consistent, etc.. I love the idea of asking folks to do a screencast if they have the tools and are comfortable doing that.
When I test with larger groups, I want to see if the strategy is moving the needle. We have a testing package we use to measure before and after snapshots of abilities and perceptions. It's pretty low tech. All paper based.
The pre-module survey and assessment contents vary based on the context. Generally we're looking for the participant to describe how they feel about a particular topic before hand, how much they know, what their understanding of rules and consequences are, etc.. This is brief as we don't want to over-influence the experience with prep, but we do want to get some sense of where folks are before. We also assess various tasks on paper ahead of time and extend that capture with a confidence measurement (How confident are you in that answer?)
We offer a similar assessment after the module experience that measures similar task contexts and asks the same confidence questions. This gives us a general idea whether the module did what it was designed to do. These tests are NOT part of the final deployment, though we do try to put L3 measures (are you using this on the job and can you accomplish the task at some point in the future) to evaluate how long the results last. If the L3 measures are off the mark and don't jive with our user test results, we make an adjustment or add something a little different to span the gap. It's all you can do, really
I like the idea of watching your learners. It always amazes me what I see just by observing. Often it's just a small tweak that makes the difference.
I do ask my testers to go down every path--get questions correct and not, go down all branching, etc.
These are awesome recommendations! I like the "watching" the learners idea too. I did that for usability testing one time and it was a big eye-opener. I'll combine that with some form of open-ended feedback option.
Should have mentioned this after reading Daniel's suggestion. We put together a set of Acceptance Testing Checklists. You may be able to extract a few things from this list of factors.
http://www.uscg.mil/forcecom/training/docs/training_SOP7_Sep11.pdf#page=163
I think it really depends on what you want from your beta testers. In my experience, if you are using non-eLearning people, they're unlikley to point out any major/minor design/technical issues unless you ask them VERY specific questions. They just don't know any better. In most cases, the small details we'd like them to point out, they won't even notice.
Typically, if I'm having my SME(s) look at the content, I'll ask them to ignor the "look and feel" of the course and simply focus on content accuracy. Is everything accurate? Is the course conveying the complete message that you want? Etc.
If I'm having a group of end-users from the targeted audience look at the course, I'll simply ask them to give general feedback on the course. With this group, you'll likely have to ask a lot of questions to get the right info from them. For example, was their anything you liked/disliked about how the course looked? Do you feel the content helped you to meet the specified learning objectives? Etc. Again, unless they've reviewed eLearning many times in the past, they won't know to give you this type of feedback.
And lastly, if I'm looking for feedback on how the course looks, it's structure and/or design; I'll have a group of my peers look at the course. They'll be the ones who really understand what you're looking for in terms of feedback AND they are the best group for this type of review.
Hope this helps! Good Luck.
-Tim
I recently beta tested a course I developed by having two people complete the course while I watched. This works for me because I am in a corporate setting. I was able to observe navigation issues as well as measure engagement. I did not interact with the participant while they completed the course but did interview them afterward. Of couse I had several others complete the course without me and then I interviewed each one. I highly recommend observing if you can.
Great point Marty...
I think observing users take the course can have a ton of benefits. Never thought of that. Do you preface anything for them before they start (i.e., act like I'm not here, interact with the course as you normally would, etc.)?
-Tim
Yes Tim. I ask them to act as if I'm not there. This also allows me to time the course instead of estimating.
In a previous project, I asked testers to narrate their activities. The narrative helped cover the areas when the user seemed confused but we couldn't tell why.
Not all testers were comfortable narrating, and some were better at it than others.
That's a great idea too, Judith. Really great insights, especially if you can record their screen and audio simultaneously.
Thanks for the great ideas. I'm getting closer to testing and feel better knowing that I have some best practices to use. I made my first course far too involved, so hopefully, there won't be a ton of technical things to fix.
Love this community! Thanks for the great ideas and feedback. This gives me a lot of confidence to do something I have not done before and be really confident about my work.
This discussion is closed. You can start a new discussion or contact Articulate Support.