Determining Behavior Changes

For our resident training offerings, we consistently measure level 3 transfer to the job with surveys. Two surveys are deployed several months after the training event. One survey to the attendee and another to the supervisor. There are some examples in our Evaluation SOP that might be helpful:

http://www.uscg.mil/forcecom/training/docs/training_SOP3_Apr11.pdf

For our L3 surveys we are more concerned whether the member performs the task (quantitative) than how well they are able to perform. The assumption is that either they can do it or they can't (or in this case, do or do not.) If they don't perform the task, we want to know why. Supervisors have the opportunity to provide feedback for a task set, so we might capture some qualitative data.

This data is used for job design and influences job performance qualifications. The TPO's / EO's are tied directly to the JPQ's so this may influence course content and execution but those effects are downstream. We've been doing things this way almost across the board for almost 20 years. It creates a nice feedback loop for tuning vocational preparation, training, and resources.

The measurement and determination of the behavioral changes can be done following the different stages of changes including Precontemplation, contemplation, action, Maintenance. Earlier the research has measured the number of cognitive behavioral markers that have been used for identifying various behavioral changes.

Thanks, guys! Steve, I really appreciate you sharing the Evaluation SOP - it was very helpful!

Keepin' the joy,

Patti Bryant

Hi Patti,

I am in the midst of a project that aims to provide practical help to organisations with this sort of problem. It's not trivial - masses of dollars and pounds are wasted on training that does not translate into change, and organisations' senior management cannot properly manage this because Level 3 change is poorly evaluated, or not evaluated at all. The work is focused around the following premises:

1. Behaviour can change directly as a result of training, but where a significant change is needed, involvement by and ownership from

trainees line managers is usually critical to coach new skills into practice.

2. The critical skill of coaching and development of staff is often found to be the lowest rated competence in leaders.

3. Despite not being done very well (or not at all), level 3 evaluation is arguably the most critical of all for our future productivity.

4. Data around training management is typically held in a different place (i.e. in the LMS) from behaviour evaluation (which is in the performance management system). These systems have different owners and often do not interrelate.

5. Any proposed solution or improvement to this has to be perceived as simple, practical and couched in business language.

One of the things I am piloting with a multinational is a 'new deal' between L&D and management. There's several parts to it: branding a 'stairs' model of learning, in which a training or elearning course is only the second stair of four; A 'deal' - if we provide you with training, you undertake to follow our guidance right through a learning process. It's being underpinned by an extension to an LMS:  As a learner you will receive a timed notice to complete 3 objectives prior to training (which are recorded online). Immediately post training you and your boss will get a notice to set a plan to put the key things learned into work projects/practice (also recorded online). 3 months later your boss will get a notice to do a simple evaluation of these plans and grade them as fully achieved, partically achieved or not started (also recorded online).

Another component will be something which digs in to the coaching skill levels of leaders and then provides context-related elearning.

I have demos and videos in preparation. Delighted to show you or anyone else when they're ready.

John,

Thanks for sharing! Love the "deal" concept where you basically keep track of the learner's progress by the notices.  It definitely coincides with the "learning isn't a one time event" concept. 

Keepin' the joy,

Patti

Hey John

This sounds wonderful - I would love to implement this system in several places I am working with.

Perhaps we can talk offline at some point?

Bruce

Hi John, 

I'm brand new to the community and elearning...have spent days browsing the forum and saving the awesome resources posted here so I can educate myself. Your concept of creating a deal between L&D and management sounds like something my L&D manager would love to explore. Any chance you are ready to share the demos and videos you mentioned?

I did promise to share some things with the Forum....I have put together some aspects of what I am doing in this presentation:

https://dl.dropboxusercontent.com/u/64840770/Working_and_Learning%20output/story.html

Please let me know if it doesn't run for any reason.

I would be pleased to continue discussion on the Forum or by email.

John.a.