Inevitably there will be a handful of learners who get stuck in and are unable to finish a module. Finding out why may be straightforward or it might be an exercise in looking for a needle in a haystack. At what point do you stop putting resources into figuring out what the problem is? What is an acceptable percentage of users getting stuck or having problems with the module? (Of course the ideal is 0%, but we are at the point of diminishing returns and need to decide when to let go.). I know that the acceptable percentage will vary but it would be great to get a range of what folks experience and tolerate. TIA!
In my experience, the number is effectively 0 unless you're dealing with unusual circumstances. At my first ID job, we had 30,000 online students. I can't remember a single instance where I heard about a student getting completely stuck so they were unable to finish a module. Having technical difficulties is one thing, but being unable to finish a module is a significant red flag that indicates a major problem. If you're delivering online courses to places with uneven internet access or where poverty affects the available hardware, that's one thing. In corporate learning not being able to finish a course is a catastrophic failure that needs to be addressed. It's not inevitable; it's highly irregular.
What are the steps you have taken so far to identify the problem?
Zero is not always possible of course. So the trick for me has been to understand what the issues are, not just the percentage threshold.
I've found the most effective strategies is to focus on the "why"...
Create/keep a list of the standard barriers to completion. It's a living list of course, so you will add to it as more problems surface.
For each issue type, research a resolution (or workaround)
Attempt to quantify (or at least extrapolate) the potential number of users impacted. Apply a percentage of the overall to this number.
Make a business decision on the value/ROI of trying to address each barrier/issue as compared to the importance of the topic and % of users impacted
This approach is easier than it sounds. And you may find that the vast majority of your barriers to completion fall into a couple a recurring issue types. With a quick couple of questions you can identify which issue it might be, and by having an established resolution or having made a conscious decision to not address that particular issue, you can handle things efficiently.
Back to your specific framing of the question..... 3% of learners not able to complete is unacceptable if you can identify it's a simple browser setting easily changed, while 10% of learners might be just fine if you can identify that the particular course doesn't work with their brand of device and would not be cost effective to address right now (business decision).
We have been keeping a running list...and a handful of folks reporting problems are getting stuck in different places (usually in relation to an interaction - completing a quiz or survey) and the "stuckness" is not reproducible. When they get stuck they cannot progress at all.
We have done a number of things:
a lot of pre-testing (at least 10 times for each module with 8-10 users with different computers, OS, browsers, and internet bandwidth) both in ScormCloud and in the LMS...testers did not encounter problems
IT checked computers, OS, browsers, and internet bandwidth for each user with a problem (computers meet the minimum specs, but the internet bandwidth varies widely and there are some significant bandwidth issues (for example, 1.5 Mb upload & download))
double-checked to make sure the modules were published in SCORM 1.2 (as the LMS does not support 2004 or Tin Can)
reloaded the final versions of the modules to ScormCloud for those who got stuck in the LMS...no one has had problems getting stuck in ScormCloud
Our "catastrophic failure" rate is 2%.
I am a designer. I do not do the LMS/IT end of things. I feel confident that it is not a design issue because of the inconsistency of the problems and my working theory is that it is a bandwidth/module-LMS communication issue (e.g., the LMS times out because the internet is so slow and therefore the user gets "stuck" because the LMS has closed the connection).
What questions are left unanswered for you? What new questions have emerged as a result of this description? What recommendations or suggestions do you have?
First, I will be using "stuckness" all day tomorrow...consider that stolen. :)
Second, it sounds as if you are very much on the right track (especially for not being an LMS admin type!).
The communication between PC and LMS is often the culprit for these sort of niggling issues and your timing-out guess may not be too far off the mark (or packet loss or something similar). As you have surmised, Scormcloud hosting will not reveal the specifics in this case, but having your LMS vendor audit communications may....
I might go back to your LMS vendor with specific requests to review the communication logs between the courseware and LMS for impacted individuals. Most/many LMSs have this functionality even if it's hidden from the regular admin. You are going to have to provide specific examples of users instances, times, courses etc, but if you do the logs may be able to reveal what happened and why.
Depending on what they come back with, you may come up with suggestions such as logging in at a low volume time of day, or you may have to reduce bandwidth of courses, or something else. But 2% is not terrible and if the answers after this next level of analysis lead to non-cost-effective solutions, you may make a business decision to be ok with the 98%. But you owe yourself the "why" so you can know for sure.
Sounds like you are on the right track and take heart in knowing that your situation is not at all unique. Good luck!
Thanks for the language to use with the LMS vendor. Hopefully the corporation for whom I have designed the modules will have someone that can make sense of the logs. I'm officially off the clock on this project, but inquiring minds want to know the why and how to do things differently!
Based on what you have done so far, it sounds like you have narrowed it down to an LMS issue rather than an issue within the courses themselves. After all, if it was in the courses, then using SCORM Cloud wouldn't fix the issue. You've eliminated all the other options. Bob's suggested request to the LMS vendor is the right next step.
I will note that since they're having bandwidth problems, they might be better off moving to an LMS that supports xAPI. One advantage of xAPI over SCORM is that it retains progress even if you're offline, so if your connection drops or times out you still get to progress. I realize switching LMSs is a significant undertaking and way outside your scope. If it was my client, I would mention it to them as a long term solution though.
2% isn't a bad failure rate when you have the workaround of using SCORM Cloud. Having a workaround means it's no longer a catastrophic failure, just a hassle. :)
7 Replies
In my experience, the number is effectively 0 unless you're dealing with unusual circumstances. At my first ID job, we had 30,000 online students. I can't remember a single instance where I heard about a student getting completely stuck so they were unable to finish a module. Having technical difficulties is one thing, but being unable to finish a module is a significant red flag that indicates a major problem. If you're delivering online courses to places with uneven internet access or where poverty affects the available hardware, that's one thing. In corporate learning not being able to finish a course is a catastrophic failure that needs to be addressed. It's not inevitable; it's highly irregular.
What are the steps you have taken so far to identify the problem?
Hi Susan,
Zero is not always possible of course. So the trick for me has been to understand what the issues are, not just the percentage threshold.
I've found the most effective strategies is to focus on the "why"...
This approach is easier than it sounds. And you may find that the vast majority of your barriers to completion fall into a couple a recurring issue types. With a quick couple of questions you can identify which issue it might be, and by having an established resolution or having made a conscious decision to not address that particular issue, you can handle things efficiently.
Back to your specific framing of the question..... 3% of learners not able to complete is unacceptable if you can identify it's a simple browser setting easily changed, while 10% of learners might be just fine if you can identify that the particular course doesn't work with their brand of device and would not be cost effective to address right now (business decision).
Hope this helps!
Thanks for your prompt response! See my response to Bob...
Thanks, Bob!
Bob and Christy...here's what we have been doing.
We have been keeping a running list...and a handful of folks reporting problems are getting stuck in different places (usually in relation to an interaction - completing a quiz or survey) and the "stuckness" is not reproducible. When they get stuck they cannot progress at all.
We have done a number of things:
Our "catastrophic failure" rate is 2%.
I am a designer. I do not do the LMS/IT end of things. I feel confident that it is not a design issue because of the inconsistency of the problems and my working theory is that it is a bandwidth/module-LMS communication issue (e.g., the LMS times out because the internet is so slow and therefore the user gets "stuck" because the LMS has closed the connection).
What questions are left unanswered for you? What new questions have emerged as a result of this description? What recommendations or suggestions do you have?
First, I will be using "stuckness" all day tomorrow...consider that stolen. :)
Second, it sounds as if you are very much on the right track (especially for not being an LMS admin type!).
The communication between PC and LMS is often the culprit for these sort of niggling issues and your timing-out guess may not be too far off the mark (or packet loss or something similar). As you have surmised, Scormcloud hosting will not reveal the specifics in this case, but having your LMS vendor audit communications may....
I might go back to your LMS vendor with specific requests to review the communication logs between the courseware and LMS for impacted individuals. Most/many LMSs have this functionality even if it's hidden from the regular admin. You are going to have to provide specific examples of users instances, times, courses etc, but if you do the logs may be able to reveal what happened and why.
Depending on what they come back with, you may come up with suggestions such as logging in at a low volume time of day, or you may have to reduce bandwidth of courses, or something else. But 2% is not terrible and if the answers after this next level of analysis lead to non-cost-effective solutions, you may make a business decision to be ok with the 98%. But you owe yourself the "why" so you can know for sure.
Sounds like you are on the right track and take heart in knowing that your situation is not at all unique. Good luck!
Steal away, Bob!
Thanks for the language to use with the LMS vendor. Hopefully the corporation for whom I have designed the modules will have someone that can make sense of the logs. I'm officially off the clock on this project, but inquiring minds want to know the why and how to do things differently!
Really appreciate your thoughts and time!
Based on what you have done so far, it sounds like you have narrowed it down to an LMS issue rather than an issue within the courses themselves. After all, if it was in the courses, then using SCORM Cloud wouldn't fix the issue. You've eliminated all the other options. Bob's suggested request to the LMS vendor is the right next step.
I will note that since they're having bandwidth problems, they might be better off moving to an LMS that supports xAPI. One advantage of xAPI over SCORM is that it retains progress even if you're offline, so if your connection drops or times out you still get to progress. I realize switching LMSs is a significant undertaking and way outside your scope. If it was my client, I would mention it to them as a long term solution though.
2% isn't a bad failure rate when you have the workaround of using SCORM Cloud. Having a workaround means it's no longer a catastrophic failure, just a hassle. :)
This discussion is closed. You can start a new discussion or contact Articulate Support.