Formalized Review/Feedback Process

Hi Tracy!

Great discussion topic.. thanks for bringing it up!

Quick question... can you clarify exactly what your current review process is? You mentioned there are up to 10 reviewers? All SMEs? Or different types of reviewers? Do they each review it one at a time, or in tandem? Do they see each others feedback when they add their own? Is there any process in place for consolidating this feedback before it's passed on to you?

When it comes to the second part, personally I'm not sure that I'd want the reviewers feedback going directly to the client. It seems to me that it would be the Instructional Designers duty to review that feedback and decide how and what changes to implement, versus the client...  Some edits may require the clients input, that I can understand, but I think involving them in every little minor edit that needs to be done would be overwhelming for them, and I'd prefer to have control over what changes are happening as the course reviews progress.... That is just my two cents... But I would also be interested in hearing what others think about this. 

Thanks again for bringing it up!

Also, here's a few links to past related discussions that might have some juicy tidbits:

• What is your testing or QA Process?
• What's your review process look like?
• Suggestions for development process & review process?

:)

Perhaps a bit of a unique situation here.  

Quite often one of our clinical educators (client and SME) will bring me a topic to be created into some elearning format.  They and I hashout the design/development/etc.  Once we get past that and I have a prototype (close to release version) it goes out for review to other clincial educators (also SMEs).  They review in tandem and love to send me individual emails about what they want editted/changed/etc.

The process I want to develop greater is that the "client" reviews if content/material/assets/ interactions work as expected and are true to the process lived by those learning (eg. entering an order) the material.  Then the "SMEs" take another look with 'fresh eyes' to see if we are missing anything relevant to their world within our walls.

So I'm looking to eliminate the barage of email messages, the extra reinterations beyond what is reasonable (got stuck in an endless loop last year and that opened my eyes wide), more understanding by my "client" of what their role as client client and sme means for the project as it moves forward.

Oh and as for your question depending on the topic the review team can be quite small and other times far too large anyone's liking (eg. last summer's endless loop).

Hi Tracy - I had these thoughts when I read your post:

1. educating them about the impact of the endless reviews (time wasted, takes longer to get to "market", opportunity cost, etc). You could get your boss involved if you wanted to. You might also tally up the hours you spend on a project and show that to them. Maybe give it to them as an equivalent ("if I was charging by the hour, it would be $xxx").

2. perhaps a "role clarification" would be useful - there should only be one client and they are accountable for the content. Maybe a RACI chart? http://en.wikipedia.org/wiki/Responsibility_assignment_matrix. I find these are helpful to frame discussion about "what will we do if..." at the early part of the project. 

3. Open-review documentation - steer them towards google docs that everyone can see. Even dropbox allows you to collaborate within now. I use Review My eLearning, which allows me to switch on the "let others see comments".  And I send them a "here's what you are focused on for now" intro email.

4. Categorize what edits vs changes are so they understand that wholesale changes are ok when things are still early in the development phase, but after that you are looking for refinement.

5. Clearly re-iterate that there are types of review. Perhaps create a spreadsheet with categories of feedback. 

As a "vendor", I don't have this as much as is likely for internal folks, as I have to remind them that their changes are out of scope and we determine if it means enough to them to make the change, then additional charges may apply. Also, I answer to the client and they sign off, not the SME (typically).

I don't know if any of these help, hopefully others will chime in with their ideas.

Holly

Hey Tracy - I don't know if this is a viable option for you or not but you could try using a reviewer tool like this one. https://www.reviewmyelearning.com/

It is neat because it consolidates all your reviewer comments in one place and reviewers can see comments left by others so it reduces redundancy.  

I've used the free trial and am thinking of adopting it for my business but have not bit the bullet yet so to speak.

Hi Tracy,

I feel your barrage-of-random-and-contradictory-feedback pain! I've had an occasional  taste of it and it's always a waste of project resources.

Like Holly, I act as an outside vendor. I'm primarily involved in the design and development phases, so that means there's always a client-side project manager of some sort.

On the rare occasions a project manager has directed SMEs to send their thoughts directly to me, it's been easy to educate the manager on the fact that:

• (A) I'm not the person in the best position to make a judgment call when it comes to conflicting feedback about the content (99.99% of the corrective feedback I get is about the client-supplied raw content - not my design), and
• (B) I get paid by the hour and they're going to get a lot more bang for their project buck if I focus on design and development rather than making sense of the senseless barrage.

So even though I'm on the outside, the principles are the same. As you've pointed out, it's a waste of your time and effort to go through those endless loops - (Holly makes a great suggestion to quantify that for them) - and it also seems unlikely that endless loopage is going to result in better training.

You usually end up with outdated training at the end of them.

Like Nancy, I've considered implementing https://www.reviewmyelearning.com/ at times when it's looked like I might get stuck at the wrong end of the SME-feedback chute, but so far I've not had to do it. It looks like a good possible solution, though.

Best of luck to you, Tracy!

Hi Tracy, 

Great discussion! I've never used ReviewMyELearning.com but it might come in handy one day. Thank you, Nancy for sharing it. 

Like Jackie, I work as an independent. Here's what I do: 

I create a page on my Wordpress website and upload the course. (I make sure the page remains hidden from all menus, tag-clouds, etc. I.e. you won't find it unless you have the link.)

I send all reviewers the link. Usually I stipulate in my contract that there be one point of contact on the client side and she sends it to the team. When I send the link, I request they post comments on that Wordpress page. (This doesn't always work; some people aren't comfortable posting comments, or maybe they just don't have time. But I have had some good conversations that way.)

I give reviewers a day or two to look at the prototype/build version and set a time and date for a 'consolidated' review.

The consolidated review is handled remotely, at my end, through Join.Me, a web-conferencing and screensharing tool. All team members/stakeholders are invited. We walk through the elearning step by step. I collect their feedback over the phone. (I frantically take notes!)

Sometimes the SMEs discuss among themselves what solution should be put in place and I listen for the decision. Other times, I need to justify my design and why I've done something a particular way. Sometimes they have really good ideas! With the screensharing, I can often show the team right in the meeting a feature or element that they're discussing. 

This consolidated review in a big web-conference meeting means everyone is there for the conversation. I reiterate my action items and any other big decisions at the end of the call, then get to work making the edits. 

I hope this helps! I've had pretty good luck with this system. Sometimes getting everyone together is like herding cats but it does eliminate a lot of the back-and-forth, endless emails. Good luck to you! Let us know how manage your next project. :)

Great discussion. I can really understand your situation and the pain you are suffering through because I  suffered a lot. When I was designing an e-learning course on SAP (ERP system). I have had same issues with SMEs, especially when they were more than 1.  Each and every person has a different eye to review my content and sometimes I can't understand what are they talking about. 

The best solution to get rid of this irritating situation is to schedule a meeting with all of them at once. Invite your client, reviewers, SMEs and then review your course.  I found it much better because except graphics and layout if anyone has any view in terms of content, assets or materials. The other person sitting next to him will most probably object it. The conversation between them keep going until they come up with the certain ground rules. You have to list down what's being discussed, and their comments (note their names too), and send a copy via email after meeting.

Instead of sending an email and asking for reviews separately. I think it's better to gather them in one room and then review your course. 

This works for me and I hope, this will help you too!!!

Regards,
MH

Good point about the let's-all-get-in-the-same-meeting-and-hash-this-out approach, Jane and Mohammad. I've done quite a few of those over time. It can be great. Two potential issues to keep in mind:

1. The weak links are the people who either don't show up or show up but haven't fully reviewed the material (except during the meeting, so they're rushed, can't do a good job of it, and haven't heard a word anyone else has said.)
   
   IF you have team members who aren't participating fully it can lead to yet more changes, issues, and meetings down the road, depending upon the importance of the information that person holds and/or their place in the pecking order.
   
   When it's the VP of Whatever-It-Is there's rarely someone with the authority or willingness to say "too bad - you missed the meeting and sent us no feedback so we're forging ahead without you."
2. The other issue with an all-hands-on-deck meeting is documenting what's been agreed upon. Jane mentioned she has to frantically take notes, and that's the problem. Somehow things have to get documented and clarified, and when you're frantically writing and your attention is split between writing and listening, it's likely you'll miss things. 
   
   There's no substitute for the clarifying and detailed verbal summary of "what we've all just decided here" at the end of the meeting (hoping everyone is still present and paying attention), followed by the emailed version to all so it's documented. (As Mohammad also does.)

Hi Tracy,

I hear what you saying! It can be very hard to get agreement from all reviewers. Having a single 'approver' on the client side is a great way to handle it.  There are a few ways to handle that. However you are collecting the feedback let the reviewers discuss it and have a field for someone to mark the comment as "approved"; meaning "Dear Developer, please take action".

We also suggest sending a checklist of sorts to all reviewers that helps point them in the direction of what to look for in the course. You can tack on to that document something like "After you submit you comments Doug will approve them. Doug may reach out to you discuss them. "

There is much debate about whether you should let reviewers see each other's comments. Some say it will taint a reviewers perspective and they will jump on the bandwagon to bash a slide. In my experience it's good to let them see all the comments. They might chime in with a "me too" or "ditto" but most reviewers will not gang up on you :-) And it cuts down on repeat comments. 

We tend to do a 'functional' review within our team before we send an invite to SMEs/Clients. This way we can knock out the navigation and interaction issues before the client sees it. We don't share those comments with the client normally. 

Sometimes design by committee is unavoidable - particularly in a healthcare setting. There are a ton of great ideas and best practices in these threads, but perhaps it's also worthwhile to consider changing up the process so that the initial design meeting with the client and SME also includes the clinical educators. That might be difficult to do with everyone's schedules and it may not be a worthwhile undertaking for every course, but such a meeting could be helpful for getting all the parties to hash out their preferences and perspectives up front. 

Thanks Nancy.  I had looked at this in the past, but forgotten about it.  It seems a great tool.  Unfortunately even at its small price point it' still more then I am allowed to budget....for now.

It's a tool I will keep in my mind though.

Excellent ideas here.  Most of these points, as you stated, are formalized in my mind.  Now it is a matter of educating them (the clients/smes) on each of these items.  Most of my clients are repeat clients, so hopefully sharing this kind of information once with them will be all that will be required to begin to get the impact I'm looking for.

Thank you kindly for these ideas.

Hi Jane:

Great ideas here as well.  I do post our prototypes onto our learning sytem directly and then hide from view except for the reviewers.

Typically I also give a few days for feedback, but have been finding my plate getting more and more full.  So the endless emails coming in about the review is what I'm limiting.

I have in the past done what you suggest, except I had everyone in the same room.  This is a great way to get everyone's opinion and at the same time listen in on the reasoning for "X and Y".  Sometimes it will give me great ideas how to represent some material.

Here, at a high level, is how my endless encounter when down last year and why I'm becoming extremely more mindful of process.

• major incident occurs
• policy/procedure/form developed my org
• education required regarding incident/how to handle/what to now do
• committee formed to do this with 5 groups of key stakeholders
• group A is tasked with developing cirriculum
• group A vets material through committee
• committee agrees on content
• group A discusses design/development with me
• we go through a normal process of design and development, 
• group A reviews and tests course
• group A asks committee chair to now review (okay...seems a good point)
• committee chair wants to show it to committee (okay let's show them what we have made)
• committee now decides on new changes to content (sigh) (okay wish they had thought of this before)
• changes made, group A and chair review again (should be done at this point)
• group B now says they need to relook and reevaluate the content
• group B has major edits (some are indeed important, but these should have come up 9 bullets ago)
• edits made
• group B wants to review again (group A & chair review again)

I lose track now after that. It may have gone through the last 4 bullets a second time and then once more back to the committee.  Then to top it off they decided that the wording of the "main subject of course" wasn't right and changed it.  That particular wording/phrase needed to be edited through out.  Thank goodness for Find/Replace.  That got most of it.  :)

It seemed endless.  Was the first time in 9 years I had come across this kind of re-re-re-review process.

Many, many lessons learned from that and hence my quest to become even more mindful of roles, responsibilities, process, etc.

Thanks for all the great ideas.

Please keep them coming if you have any more, I am so open and grateful for them all.

Tracy

One of the things I've learned is to make sure that you know all of the parties who will be signing-off on the project. I had an instructor-led course that I was developing that went sideways a few times as different business units argued about the course material and I had to be more of a facilitator than a designer. Just as I figured that we were at the end of the design and development phase a new player appeared and I had to do another round of changes and sign-offs.

That was exactly what happened Cary.  A very important lesson for any ID to know and be aware of before starting any project.

Hi all - just wanted to let you know that I do use https://www.reviewmyelearning.com/ and it is worth every penny. It's made content review much easier. I could even add comments on the slides myself as a reviewer (highlighting certain design choices or elements that I'd like them to pay particular attention to) if I wanted. I add developers to the site, they can see the comments and can ask questions about the comments to me or directly to the reviewer or provide insight about why something can't be changed or if the change is out of scope. And, I can export to spreadsheet to keep a record of changes, which helps with scope creep.

Holly

Hi, Everyone, I am new to this community, but wanted to share our experience. Technical solutions we use are either

google docs,

http://www.invisionapp.com/

or https://trello.com/

in reality I believe no technical solution can help if there are 10 reviewers, it all comes down to how well the client knows what they need, and how well they are able to communicate it :). One method we use if it is absolutely messy review, we ask if all of the rieviews have been sent, and only then arrange a meeting with the decision makers to go through them, and only then start working on improving the project.  I totally get your pain, it can drive anyone crazy :).

That's great, Holly! Always good to hear the details of first-hand experience. It's still at the top of my list if I decide to go down the road of using a tool to consolidate reviews.

Our team also uses reviewmyelearning, it is very good for getting feedback on the course.

Thanks Karlis.  That's a good idea to collect reviews and discuss with decision makers.