How do you measure the ROI or value of training?

Hello helpful community peeps, 

I'm looking to find out how you measure the value of the training you create/deliver, and any tips or tricks you might be willing to share with others about that process.

Post-course evaluations are a really nice way to gather feedback and peoples opinions (opinions being the key word) but as training professionals we want to be able to show the actual return on investment or the dollar value of the training solutions we deliver. 

I'd love your tips for how you accomplish that!

8 Replies
Sean Delaney

Hi Nicole,

Great question, I think it is something we all deal with. I have worked with learning evaluation teams, more from the Instructional design side, creating objectives, assessments and conducting observations and then handing over the data for analysis. I’ll share some of my learning. As a reminder Kirkpatrick’s levels are commonly used to define the depth of evaluation that will take place:

Level 1 Reaction: To what degree participants react favorably to the learning event.

Level 2 Learning: To what degree participants acquire the intended knowledge, skills and attitudes based on the learning experience.

Level 3 Behavior: To what degree participants have learned during training when they are back on the job.

Level 4 Results: To what degree did your targeted outcomes occur.

Level 5 Return: on Investment, measure the costs saving of your business results against the cost of the training.

 From my experience some of my tips are:

1.       You need to define what level of evaluation way before the design of the program starts, as there is data to be collected before the training/course.

2.       When creating your learning objectives they should have evaluation in mind, i.e. be clear how you plan to measure each of them after wards.

3.       Level 2 can be measured via tests in the course or a certain period afterwards. Often the same pre and post test (or at least question bank is used) so that the results can be compared. Doing the test as part of the course can be logistically easier to ensure you capture all results but doesn’t account for how long the learning has been retained over time.  

4.       Level 3: Can be achieved via observation or review of data. Note for Level three it is important to have the before sample in place. So for example in a customer service example you wish to improve the experience of shop assistants at a counter. Let’s say the learning objective are to improve the customer greeting, the up sell and closing.  You literally observe a sample of interactions and rate the experience for each section on a scale you have created. You repeat the activity after the training is complete to compare the results.

Sources of data may exist already depending on the scenario at hand. How you decide to measure a behavior is known as your  ‘Leading indicators’.  There are ideas for these online for various scenarios.

5.       Level 4: Results can be achieved by reviewing statistics and performance reports before and after the training to see the results.

6.       There is a common discussion on how do we know the training or learning experience had the impact as it could have been many number of things. This is true but there are techniques to isolate the training impact by measuring other activities that are occurring at the same time. For now, let’s just say it is important to consider this.

7.       Learning evaluation is very much about data analysis, whether utilizing existing data or creating new data from qualitative research. Having someone on board that understands statistical analysis is important to create meaningful results.

 Measuring the results of training or learning experiences is a whole skill set in itself and really should be considered a separate project to the course design and delivery itself. I have experienced a bias to underestimate the time required to complete meaningful evaluation. For level 3 and above it should be a considerable percentage of the overall project time. If you are limited on the time, as we often are, consider focusing on level 1 and 2 and implementing the learning from these as the priority.

 

Bob S

So a couple of more thoughts to go along with Sean's thoughtful response....

Level 3 - The other common way to do this is to survey supervisors regarding specific behavior change they have observed (or not).  This is a cheap and easy way to get some data, even if is admittedly less precise than other metrics. 

Level 5 - A starting point is to understand if your business has a sense for the actual value of their worker's activities.  For example... Most hospitals know the value of a bed on a daily basis, but do they  have a value for each touchpoint in patient care? Restaurants can tell you the value of a table, but what does upselling wines really represent in delta dollars? If your business does not measure the value of these sorts of activities now, then trying to measure them for training purposes may be a bridge too far.

That being said, if you do want to take it on, this sort of analysis is often best done in partnership with the reporting arm(s) of a business entity rather than coming solely from the L&D function. It will be more credible, accurate, and will garner less resistance from stakeholders. WAGs are just fine for many things, but if you really want to head down this path of hard ROI justifications, then you must begin by knowing the starting values before talking about change in value.

Alexander Salas

I concur with my fellow Kirkpatrickans here ;  )

Levels 1 and 2 pre and during training phase- after that is not that helpful to measure knowledge with a test.  Today's worker is an information resource finder rather than a knowledge storage machine ;  )

Level 3 - Remember that JTA we did in the analysis phase? Oh don't have one? Well, we need some competency checklists then. I give one to employees so they do a self evaluation of knowledge-skills transfer to the job.   Supervisors are also given one to provide performance feedback according to the learning objectives which should match the original business needs.

Level 4- Increased sales? Improved workflows? Improved customer satisfaction? Less errors? It's all about metrics and correlations to the previous levels. 

Level 5- Ain't nobody got time for that ;  ) at least not in a contact center environment and especially if there are no distinct budget allocations for training.

Jeff Kortenbosch

I pretty much agree with the above. If you're not measuring anything yet start with a satisfaction rating that has some meaningful questions. Probably after the course and a little while later (1-2 months). That way you at least have something to show to your management which is better than nothing at all.

After that it's all about performance impact for me. I havent figured out how I want to do those exactly but it's like Alex says; determine the performance metrics in the analysis phase find out where you are now and set desired results. Afterwards test if you've improved and have met the desired results.

Kate Salvan

There are no accurate means of Training ROI calculating as there is a number of factors to take into consideration such as training implementation goal (cut of overall training costs, employee qualification improvement and its future benefits), industry type and much more. However, there is a way of receiving an extra economy from the cost per learner point of view: the more learners you have the lower price per learner is.
To maximize effectiveness, training implementation should be started from a small group of devoted and enthusiastic people and then little by little proceeding to a full-scale usage. The reason why such kind of approach is the most acceptable is a chance to reveal and fill in the gaps of training in terms of a small group, excluding large-scale mistakes by taking into account feedback and constant effectiveness monitoring by numerous means of LMS, such as reporting system and different LMS level access.

Nicole Legault

Thank you so much for this thorough response, Sean. There is a lot of valuable tips and tricks in your answer.

When it comes to Level 3, I do think one thing that can cause an added layer of complexity is being able to isolate the training as the actual cause of the performance improvement, when so many other factors could affect performance as well, as you mention in your point #6.

For example, you're providing training to a sales team and you're intending to measure the level of sales as an indicator of training success... but what if during the time the training was designed, delivered, etc. there was also a sales promotion occurring ... that could also affect the metrics. Any tips or suggestions there on how you are able to isolate the training as the cause of the performance improvement? What are the techniques you use to isolate the training impact?

Nicole Legault

Bob S - That is such a great point about "evaluating training being a bridge too far" if a company is not able to actually correlate on the job tasks to specific dollar value, performance indicators and business goals. I recently wrote an article about using KPIs to measure the value of training and that's all fine and dandy, but if a company does not have KPIs or metrics in place (which I think many actually don't...?) then how can you really do this to begin with? 

What are WAGs?