E-learning Vendor Evaluation Criteria

Hi Ashwin,

Great question! Is this criteria something you're going to use once? Or will you need to reuse it? If it's just for this one time, I think it makes sense to adapt the criteria to the specific exercise you requested from the vendors.

I had a look at your criteria and I think you've got the right idea in terms of what you're looking for, however I think I would personally use an Excel spreadsheet instead of a word document. That way you can set up all your criteria alongside the different vendors, making it easier to compare their scores.

Another thing I noticed is that there's a lot of subjective wording in your document, which may make it hard to evaluate the vendors objectively. For example, you wrote "user friendly navigation." How do you decide whether or not the navigation is user friendly? If 2 interfaces are user friendly, how do you decide which one is more user friendly than the other?

I would write definitions for each of your criteria that are as objective as possible, so you can rate each of them and come up with a total score. You could also decide to weight certain criteria if you think they're more important than others.

For example:

I hope that helps! Let me know if you have any other questions.

Thank you, Allison! Yes, we are eventually going to populate these criteria into an Excel sheet and weight it along with the Vendors just as how you have mentioned above. 

I agree that some of the criteria were subjective. I din't want it to be very specific because there is another section below the technical criteria that has the scores, similar to what you have mentioned (what does a 5 or 4 denote for that particular criteria). 

But I think I should rephrase some of the wordings. I am also unsure if I should include the Project Management part as we are only going to view the demo samples and won't get too much insight into their PM skills with that.  

What do you reckon?

Cheers

Ash

If you aren't going to have any insight into their PM skills, I agree it's probably best to take that off. However, it's too bad that you won't have that information! Is it too last to ask them to provide it to you?

When we received their initial proposal there was a page where they briefly spoke about their development process and quality metrics. Maybe we could use that to gauge their PM skills?

I guess it depends on how detailed it is and if you think you're able to get a good picture of their PM skills from it.

Hi Ashwin,

Looks like a great start.  And Allison stole my first tip about creating a scale for rating. :)

So another tip might be to include Evaluation Strategy in the criteria.  How is the vendor going to ensure the trainings hit the mark and update them as needed?

Ashwin - some additional thoughts:

1. differentiate on the creative approach the vendor took to the design and thought process that they used

2. did any one propose an interactive video in which the learner can interact with the video?

3. look at the video demos on various devices and see if there are any differences, quality, etc.

4. look at the load time - did any vendor provide pre-loading or buffering to reduce the learner's wait time?

5. when you asked for 3 different levels of complexity, were you looking for content/visualization complexity (the wow factor) or was the complexity in the instruction design aspects of the video? This should be a selection criteria issue.

6. Asking a vendor to produce three videos for free might be a heavy lift for some vendors. Look at their approach, the types of questions or points of clarification that the vendor asked (judge them on the quality of their questions), how responsive were they, pretend that you were the learner, when you saw each video for the first time, which vendor and/or video impressed you the most (your reactive and engagement factor), etc.

7. which video concept made you become more engaged via their storytelling skills.

Regards.