I'm interviewing for a new job next week and am wondering how YOU measure the effectiveness of the courses you create and place on your LMS. My company deals in software, so 99% of my courses are videos or simulations for customers teaching them how to use the software product.
What metrics do you collect about your eLearning?
How do you measure the impact (decreased calls on the topic to your service desk?)
I'm open to ideas and eager to hear how you measure effectiveness.
I suspect this might be a tricky question for some people as there's sometimes a gap between the metrics we know would be useful for measuring impact and the ones people in learning roles are actually able to collect.
For instance, early in my career when I was creating bank compliance training we collected completion data and test scores for learners. This was good for meeting compliance standards and protecting the company in case of an audit. But it wasn't as useful for checking the impact on what the regulations were trying to protect against. My team would have loved to have data on things like if there was an increase in how accurately bank staff recognized signs of money laundering when opening new accounts after the training. But information like that often wasn't easy to access, wasn't collected that way, was too hard to separate from other variables, or all of the above.
In an ideal situation, though, I like using the business goals for the project to determine what the metrics of success are and what supporting data to capture. And sometimes, especially if you're close to where the data is being collected, this can be easy to do. One example is when I worked on conference programming. We had a goal to boost the number of strong session proposals we received, so we created training on proposal writing tips and tricks. In that case, the metrics we collected on the training and the measures of impact were the same: what percentage of proposals made the cut before the training launched versus after.
1 Reply
I suspect this might be a tricky question for some people as there's sometimes a gap between the metrics we know would be useful for measuring impact and the ones people in learning roles are actually able to collect.
For instance, early in my career when I was creating bank compliance training we collected completion data and test scores for learners. This was good for meeting compliance standards and protecting the company in case of an audit. But it wasn't as useful for checking the impact on what the regulations were trying to protect against. My team would have loved to have data on things like if there was an increase in how accurately bank staff recognized signs of money laundering when opening new accounts after the training. But information like that often wasn't easy to access, wasn't collected that way, was too hard to separate from other variables, or all of the above.
In an ideal situation, though, I like using the business goals for the project to determine what the metrics of success are and what supporting data to capture. And sometimes, especially if you're close to where the data is being collected, this can be easy to do. One example is when I worked on conference programming. We had a goal to boost the number of strong session proposals we received, so we created training on proposal writing tips and tricks. In that case, the metrics we collected on the training and the measures of impact were the same: what percentage of proposals made the cut before the training launched versus after.