How do the rest of you handle the review and edit process? Providing e-learning for a large organization can lead to reviews and edits by numerous people. Just when you think they're ready to accept it, there's another voice to be considered. Do you limit the process to one set of edits and then bill by the hour for any extra changes?
I've done that when writing articles and reports but I didn't write the e-learning contracts in that way.
I usually try and scope the project to include "...one round of QA by named reviewers x, y and z".
Any extra time incurred by eLearning groupies gets billed at an additional agreed hourly rate. Even if the groupies pass their comments to x, y, or z, it's usually easy enough to spot.
As usual, it's all about trust, agreeing upfront who will be involved, the benefits of sticking to that arrangement, and the penalties for infringements. So long as you have a good working relationship it usually works.
Usually worth bringing the scenario to light in the the planning/quote/pre-quote phases, and then when you have an agreement write the contract in those terms, even if it is just a "Confirmation Email - Rules of Engagement".
Others usually get involved because they want to ensure their "bit" is not forgotten....which goes back to scope. If it's In Scope, it's additional and chargeable, if it's out of scope, it gets put it at additional cost, or into v2 at the 3/6-monthly update (chargeable).
I also like to get the names and responsibilities of any SMEs/reviewers up front too - just so we collectively know who's in and who's out. we then agree how the reviewing will work and kind of ring fence that in. Getting commitment from these people when the time comes can be another thing altogether, e.g. person A takes the lead while B and C can't find the time for 2 weeks and then suddenly have a bunch of new ideas to throw in.
So long as boundaries are established earlier rather than later, you should have at least some "authority" over how/where to charge and delegate etc.
Similar to Bruce and Simon. We always have a project "owner" and that person has the final say. This is usually the person who came to me with the initial request. Sometimes it will be two or three people, but it's usually a single person. So much of the feedback received is subjective that we found the need to focus that down to a very limited number of decision makers. The first cut of a project goes to the project owner(s) and then they decide who it goes out to. It's their responsibility to aggregate the feedback and decide upon the changes they would like. It's too easy to have it spiral out of control. I always provide a hard date when the feedback is needed by. If they choose to come in with more changes after that, it will impact their delivery date and we let them know this right up front.
My boss is always keen to remind us that feedback is just that...feedback, and that doesn't mean you have to act upon it.
With my position, I don't bill for hours so don't have that stipulation to put on my reviewers. I always try to get a member of the target audience who isn't involved in the content to review as they have a step back and provide valuable insights. However--I just had a project that was 'FINAL' and the SME went back to have multiple others review and decided to make significant changes after that. We had a good conversation that this could push the release date out (of course it didn't....but at least I said it and felt better about it).
Great discussion! Sometimes if you have lots of reviewers it helps to have a "point person" for each logical group or department who will review the reviewers' feedback. That person can then "roll up" their group's feedback and decide what's worth relaying to you & what isn't.
I like the notion of agreeing in advance on who's going to serve as the decision-markers/reviewers. The course development agreement in the downloads area is a nice freebie that you could adapt for your own situations... it includes sections for reviewers & stakeholders, so everyone's clear on who's who, in advance.
There's a lot of good feedback in here. Especially for what to do for future courses. I'm guessing this is freelance or for an external client? Are you trying to find a solution to a situation you are in right now? Possibly have a conversation with the project owner. I have often found that if I discuss a challenge from the perspective that I'm concerned about the state of the project they will often help you with the solution (sometimes even hand it to you on a platter!). Maybe try laying out the situation, explaining that if something isn't correct you are happy to add that into the edits, but as Kate said, this ongoing editing and approval cycle could really damage the timeline. At what point would they like you to stop accepting tweaks and additional edits to protect the deadline and a potential need to renegotiate the project terms.
The next time this comes up, in addition to the great tips above, here is another. I always require final sign off from the highest level of stakeholder I can get while the project is in script form. I tell them during our initial meetings that once I begin the graphic design portion we will only make changes that are absolutely necessary because it will greatly affect the timeline. I also send basic design and prototype pieces to the stakeholder early on to get that approval and avoid major changes further down the line. I have yet to come up against a disagreement here because I'm ultimately just trying to protect the project (my sanity is in there too, but I keep that to myself)
Great advice in here!I'll add my two cents (geared primarily to future endeavors):
I always scope one review cycle (i.e., round of edits) per deliverable.If I've worked with the client before and know they like to "fine tune" then I'll scope two review cycles per deliverable - and price accordingly.For e-learning courses/modules that aren't too long/complicated, the deliverables are usually a design document, a framework (sometimes), a storyboard, and the final programmed course.I'm pretty clear in the SOW that additional review cycles could result in a delay and/or additional fees.I almost never actually act on this, but I like having the comfort ofknowing that I can if things get out of control.
I make it clear that each deliverable is dependent on the one before.So for example if the design doc gets edited and approved by three reviewers, then ten reviewers give edits on the storyboard, that's cool with me as long as those edits aren't things that should have been brought up during the design doc review (e.g, additional objectives!).
If there are more than a few reviewers, I ask the client to assign someone on their side to "collate" the edits - I've found that person is often able to sort out the valid edits from the nonsense.=)If they want me to do this, no problem, but then someone on their side has to take responsibility for resolving any conflicts.
I really like the idea a couple of folks have mentioned about identifying reviewers up front… I'll have to try that!
Not sure if you found a solution for this… We were havingissues getting good reviews from SMEs, stakeholders, etc. so we created our ownsolution. We can upload our courses tothe cloud and send review invites to people. All the comments are collected inone place. Plus it has Screenr built into it. It’s been working really greatfor us and we decided to open it up to others.
Just a few idea, since much good advice was given above:
1. For the next project you work on, get a good idea in your mind in how you want the review process to go. Clients have needs, and so do instructional designers.
And here's a common metaphor around reviewing that maybe be helpful (heard it first from Patti Shank): "Building an elearning course is like building a house." Sure, you can decide after the fact that you actually wanted five bedrooms instead of four, but that's an inefficient and expensive way to build something.
I've found it's very helpful to have in the statement of work that you require one key or main contact who will be responsible for providing you feedback / and who is responsible for approvals. Then set clear points in the project where an approval is required to move forward, and any backward movement requires a change order.
It's worth establishing up front who will be involved in the review process AND asking if there is anybody who will be added at a later date. I mention this because people may be absent through maternity/long-term sick leave and suddenly make an appearance 3-4 months down the line asking for a bunch of re-writes.
Ensuring one named individual is responsible for collating/aggregating all comments can also be invaluable. It's easy to get into a habit of continually accepting a load of ongoing emails from a whole bunch of people (most of whom you probably don't even know) each of whom have their own ideas on what needs to be changed ... so you process them one-by-one ... while asking others to confer with others ... and before you know it, you've spent half a day more than you needed to. Hand this responsibility over to the client. If something doesn't make sense, ask that one named individual to answer themselves or follow up with the reviewer and get back to you. It makes the whole process much easier all round.
Changes - I've had many projects that have been on version 26.0 despite change control.
Set up your project team
When you set up your project team think about your sign off team. Make sure that they have authority, get it written down that they will be the sign-off for the project. Also check that there will be no other 'secret' sign off or reviewers - marketing teams who control brand output often will want to have input are a good example.
Collating Changes
One of the major issues that I've come across is not collating changes - some changes requests come in every other day. I suggest that you get your client (internal/external) to collate them, review them and then send them over to the developer in one set for a change review.
Who has sign off responsibility?
Make sure that you all know who is going to sign off what and when. Also make sure that people understand what sign off means. Don't forget that you work in the industry all the time, for some people this will be the first time that they have worked on an e-learning project. Don't be surprised if you haven't described the process and then want to backtrack after sign-off
Document everything
Document every request from your client and your own team. you can then review each request and then either approve or not approve it. This provides a great audit process and also helps with the the too many reviewer process. When you put an initial next to a change and put it in a formal document people think before they request a change!
13 Replies
Hi Jon,
I usually try and scope the project to include "...one round of QA by named reviewers x, y and z".
Any extra time incurred by eLearning groupies gets billed at an additional agreed hourly rate. Even if the groupies pass their comments to x, y, or z, it's usually easy enough to spot.
As usual, it's all about trust, agreeing upfront who will be involved, the benefits of sticking to that arrangement, and the penalties for infringements. So long as you have a good working relationship it usually works.
Usually worth bringing the scenario to light in the the planning/quote/pre-quote phases, and then when you have an agreement write the contract in those terms, even if it is just a "Confirmation Email - Rules of Engagement".
Others usually get involved because they want to ensure their "bit" is not forgotten....which goes back to scope. If it's In Scope, it's additional and chargeable, if it's out of scope, it gets put it at additional cost, or into v2 at the 3/6-monthly update (chargeable).
Hope this helps.
I'll send u my invoice later
Bruce
Yep, +1 Bruce's policy.
I also like to get the names and responsibilities of any SMEs/reviewers up front too - just so we collectively know who's in and who's out. we then agree how the reviewing will work and kind of ring fence that in. Getting commitment from these people when the time comes can be another thing altogether, e.g. person A takes the lead while B and C can't find the time for 2 weeks and then suddenly have a bunch of new ideas to throw in.
So long as boundaries are established earlier rather than later, you should have at least some "authority" over how/where to charge and delegate etc.
Similar to Bruce and Simon. We always have a project "owner" and that person has the final say. This is usually the person who came to me with the initial request. Sometimes it will be two or three people, but it's usually a single person. So much of the feedback received is subjective that we found the need to focus that down to a very limited number of decision makers. The first cut of a project goes to the project owner(s) and then they decide who it goes out to. It's their responsibility to aggregate the feedback and decide upon the changes they would like. It's too easy to have it spiral out of control. I always provide a hard date when the feedback is needed by. If they choose to come in with more changes after that, it will impact their delivery date and we let them know this right up front.
My boss is always keen to remind us that feedback is just that...feedback, and that doesn't mean you have to act upon it.
With my position, I don't bill for hours so don't have that stipulation to put on my reviewers. I always try to get a member of the target audience who isn't involved in the content to review as they have a step back and provide valuable insights. However--I just had a project that was 'FINAL' and the SME went back to have multiple others review and decided to make significant changes after that. We had a good conversation that this could push the release date out (of course it didn't....but at least I said it and felt better about it).
Great discussion! Sometimes if you have lots of reviewers it helps to have a "point person" for each logical group or department who will review the reviewers' feedback. That person can then "roll up" their group's feedback and decide what's worth relaying to you & what isn't.
I like the notion of agreeing in advance on who's going to serve as the decision-markers/reviewers. The course development agreement in the downloads area is a nice freebie that you could adapt for your own situations... it includes sections for reviewers & stakeholders, so everyone's clear on who's who, in advance.
Hi Jon,
There's a lot of good feedback in here. Especially for what to do for future courses. I'm guessing this is freelance or for an external client? Are you trying to find a solution to a situation you are in right now? Possibly have a conversation with the project owner. I have often found that if I discuss a challenge from the perspective that I'm concerned about the state of the project they will often help you with the solution (sometimes even hand it to you on a platter!). Maybe try laying out the situation, explaining that if something isn't correct you are happy to add that into the edits, but as Kate said, this ongoing editing and approval cycle could really damage the timeline. At what point would they like you to stop accepting tweaks and additional edits to protect the deadline and a potential need to renegotiate the project terms.
The next time this comes up, in addition to the great tips above, here is another. I always require final sign off from the highest level of stakeholder I can get while the project is in script form. I tell them during our initial meetings that once I begin the graphic design portion we will only make changes that are absolutely necessary because it will greatly affect the timeline. I also send basic design and prototype pieces to the stakeholder early on to get that approval and avoid major changes further down the line. I have yet to come up against a disagreement here because I'm ultimately just trying to protect the project (my sanity is in there too, but I keep that to myself)
Best of luck!
Great advice in here! I'll add my two cents (geared primarily to future endeavors):
I always scope one review cycle (i.e., round of edits) per deliverable. If I've worked with the client before and know they like to "fine tune" then I'll scope two review cycles per deliverable - and price accordingly. For e-learning courses/modules that aren't too long/complicated, the deliverables are usually a design document, a framework (sometimes), a storyboard, and the final programmed course. I'm pretty clear in the SOW that additional review cycles could result in a delay and/or additional fees. I almost never actually act on this, but I like having the comfort of knowing that I can if things get out of control.
I make it clear that each deliverable is dependent on the one before. So for example if the design doc gets edited and approved by three reviewers, then ten reviewers give edits on the storyboard, that's cool with me as long as those edits aren't things that should have been brought up during the design doc review (e.g, additional objectives!).
If there are more than a few reviewers, I ask the client to assign someone on their side to "collate" the edits - I've found that person is often able to sort out the valid edits from the nonsense. =) If they want me to do this, no problem, but then someone on their side has to take responsibility for resolving any conflicts.
I really like the idea a couple of folks have mentioned about identifying reviewers up front… I'll have to try that!
Hey all,
Not sure if you found a solution for this… We were havingissues getting good reviews from SMEs, stakeholders, etc. so we created our ownsolution. We can upload our courses tothe cloud and send review invites to people. All the comments are collected inone place. Plus it has Screenr built into it. It’s been working really greatfor us and we decided to open it up to others.
You can check it out at ReviewMyElearning.com
Thanks
James
Hi, Jon:
Just a few idea, since much good advice was given above:
1. For the next project you work on, get a good idea in your mind in how you want the review process to go. Clients have needs, and so do instructional designers.
And here's a common metaphor around reviewing that maybe be helpful (heard it first from Patti Shank): "Building an elearning course is like building a house." Sure, you can decide after the fact that you actually wanted five bedrooms instead of four, but that's an inefficient and expensive way to build something.
I've found it's very helpful to have in the statement of work that you require one key or main contact who will be responsible for providing you feedback / and who is responsible for approvals. Then set clear points in the project where an approval is required to move forward, and any backward movement requires a change order.
+1 for Graham's comment.
It's worth establishing up front who will be involved in the review process AND asking if there is anybody who will be added at a later date. I mention this because people may be absent through maternity/long-term sick leave and suddenly make an appearance 3-4 months down the line asking for a bunch of re-writes.
Ensuring one named individual is responsible for collating/aggregating all comments can also be invaluable. It's easy to get into a habit of continually accepting a load of ongoing emails from a whole bunch of people (most of whom you probably don't even know) each of whom have their own ideas on what needs to be changed ... so you process them one-by-one ... while asking others to confer with others ... and before you know it, you've spent half a day more than you needed to. Hand this responsibility over to the client. If something doesn't make sense, ask that one named individual to answer themselves or follow up with the reviewer and get back to you. It makes the whole process much easier all round.
Some great stuff in this post.
Changes - I've had many projects that have been on version 26.0 despite change control.
Set up your project team
When you set up your project team think about your sign off team. Make sure that they have authority, get it written down that they will be the sign-off for the project. Also check that there will be no other 'secret' sign off or reviewers - marketing teams who control brand output often will want to have input are a good example.
Collating Changes
One of the major issues that I've come across is not collating changes - some changes requests come in every other day. I suggest that you get your client (internal/external) to collate them, review them and then send them over to the developer in one set for a change review.
Who has sign off responsibility?
Make sure that you all know who is going to sign off what and when. Also make sure that people understand what sign off means. Don't forget that you work in the industry all the time, for some people this will be the first time that they have worked on an e-learning project. Don't be surprised if you haven't described the process and then want to backtrack after sign-off
Document everything
Document every request from your client and your own team. you can then review each request and then either approve or not approve it. This provides a great audit process and also helps with the the too many reviewer process. When you put an initial next to a change and put it in a formal document people think before they request a change!
Here is a just a few things that will help here:
6 things that will help with the review process
http://www.realprojects.co.uk/blog/e-learning-development-6-ideas-to-how-to-manage-the-review-process/
Hope this helps,
Scott, great post and article. Wonderfully practical and relevant to all project and team sizes. Thanks for sharing!
This discussion is closed. You can start a new discussion or contact Articulate Support.