An Assessment Journey #2: Confidence Based Assessment in Storyline 360

Practically all organisations I initially engage with worldwide - public or private, high tech, finance, health or consulting - fundamentally don't understand the risk to their business success encapsulated within this quote :

“The greatest enemy of success is not ignorance, it is the illusion of knowledge.”

The focus of most business training projects is tackling the ignorance issue.

Obviously nothing wrong with any of that. But it does miss a really important group, a group which might include some of your very best people, people who might be moments away from making a decision for all the right reasons but, if they knew better, they would quickly realise that their intended plan is simply the worse thing that they could do.

Within the Cabinet Office at the UK I have created an online platform that we are using across 30,000 staff within the Civil Service to evaluate the commercial acumen of individuals using a complex online situational judgement assessment that I have designed and developed using Storyline 360.

The assessment poses a number of situations (i.e. "You are about to go into a Contract Negotiation and realise the key negotiator from the supplying business is actually your ex boss from a previous job").

A number of possible outcomes are proposed (i.e. "Say nothing", "Tell others in your team but proceed with the meeting"...etc).

For each proposed response learners indicate whether they AGREE, DISAGREE or DON'T KNOW that the proposed response is appropriate.

This tell's us what people know - who has the requisite knowledge to identify the appropriate and inappropriate responses.

This is where most assessments would stop - but ours goes one step further.

The assessment also provides a Confidence Score that is mandatory for each question.

On selecting whether they AGREE or DISAGREE to a response each learner is asked to score their level of confidence (on a sliding scale from 1 - 100) - literally how confident they are that their selection is correct.

This means that a scoring algorithm is able to group individuals into various categories including:

• Confidently Knowledgeable - a correct response and they were very confident that they were correct.
• High Understanding with low confidence - a correct response but low confidence.
• Misunderstanding with High Confidence - the response was incorrect but the learner was very confident it was correct

It's the identification of individuals in the final category (Misunderstanding with High Confidence) that is so valuable with this approach.

We now know who has the "illusion of knowledge".

I've developed a simple demonstration here (there are three banks of questions related to the Sun, Jupiter and Mars - it takes about 2 mins to answer the questions!)

What do you think?

https://360.articulate.com/review/content/956eeb28-b4e3-4cb8-87a6-8bea780cfebb/review