At the very least, you might want to outline a few focus questions.This will serve a few purposes:
1. It will force you to reevaluate whether you've achieved your objectives before sending for review
2. It will give the stakeholders and SMEs a starting-point for discussion and feedback
...and...related to #2... **drumroll*
3. It will -- hopefully -- prevent them from just giving a random list of comments as long as your arm, which is what's likely to happen if you don't give them a point of (limited) focus. This is because, if you don't provide focus and tell them what you expect, they won't know what to give feedback on; consequently, the default position will be to either give feedback on everything in minute and painfully pontificating detail... or... feedback on nothing at all! Neither or which is helpful to you.
May be send it via dropbox. You can also maintain an Excel sheet - Slide title and comments. Both you and your SMEs can update the Excel sheet with comments.
I agree with El. It's important to focus the feedback by reminding your reviewers what you're trying to accomplish.
Also, to Poornima's point, a spreadsheet can be used to track edits. We have SharePoint in our organization so I set up a tracking list for each course that goes for review and then reviewers can add comments, even attachments like screenshots or other documents.
In addition I always send a Word document version of the course so that reviewers have somewhere to make notes. I'll often send a link to the course out with the Word document attached to the email or available alongside the course in the LMS. I'll usually set up a meeting with the reviewers about a week after I send the course for review to gather their comments and go through the course with them. I find this especially helpful if this is the first time I'm working with that person.
I normally send a link to the course on DropBox, and a version saved as a Word Document, with "Directors Notes" and guidance, as already mentioned.
When we do the second QA, (notmally takes 2 cycles), I re-emphasize that this is THEIR course, for THEIR customers, and they now need to make a go:no go decision on whether I have imterpreted and implemented their requirements correctly. Very often an ID will be blamed for any failure - so make sure you close that particular door to them.
Thanks guys for all the good advice. I like th eidea of sending the word document and maintaining excel spreadsheet. I won't be able to use dropbox but I think I can use my team sharpoint site.
I can't find it right now but Tom did a post on embedding a google doc to collect course comments/feedback. I really like that approach and you could also use any survey tool like Survey Gizmo/Survey Monkey. All of those automatically collect the feedback in a spreadsheet and allow you to make whatever fields you want (including drop-down lists for categories, etc.)
Instead of embedding you could also include a "Comments" link in the top menu or on every slide to do this too.
One of the benefits of this approach is that you can get comments from multiple people all at the same time and save it into a single document without any of the copy/paste combining of the data.
In the past I created a Sharepoint site that had the courses loaded as well as a spreadsheet for their comments. I also included a sample of the type of comment I was looking for. In my email out, I would tell them that we (the training team) had already done a spelling/grammar/ check and that the course adhered to our development principles such as colors, graphics and things. In the spreadsheet, I had a column for my comments back to the reviewer where I would indicate if I had made the edits, or why I chose not to.
Oh yes, another thing - I ALWAYS put a big text box somewhere on each slide with the slide number. I take it out for the final draft, but it makes commenting/reviewing MUCH easier for everyone.
Thanks all for these great ways fo sending material to the SME for review.
I am still not being able to publish the course to a word document and see teh images. As far as I understadn all the images in each slide should be in the word document when publised to word? But I just get blank pages, without the images or text in them
I attached a screen shot with a simple answer even tho' I'm assuming you already clicked "Show Layers" on properties when publishing to word. Each layer will print the images with each layer labled. The text shows under the base layer image.
Yes, I did all that. I am now getting an error. I think it has to do with having permission issues. I have to contact my IT admins to figure out what setting I need in word to make this owrk.
This topic of the logisitics of SME/Stakeholder review seems to come up over and over.
In the advertising world there are lots of services/sites available that allow content to be reviewed, marked up, and the comments catalogued. Many even include review database and reviewer management features. I'm told this kind of remote review tool has become a "standard" amongst the best agencies that allow them to work remote artists/designers around the world.
Sadly there seems to be a hole for this kind when it comes to e-learning courses.
When I investigated this late last year, there were precious few options available to remotely review and comment on interactive Flash content (plenty for non-interactive Flash video). And only one I could find aimed directly at e-learning.
Seems to me there is a TREMENDOUS OPPORTUNITY for the folks at Articulate to leverage not only their authoring tool expertise but also LMS experience to offer this service...
I use my server all the time, FTP and send a link.
The problem with Dropbox is a performance one, i get comments about it being slow or not refreshing. Not good if it is the first time they are seeing it.
Not sure if you found a solution for this… We were having issues getting good reviews from SMEs, stakeholders, etc. We tried a lot of the options mentioned here, Google Doc, Excel, Dropbox etc. In the end we created our own solution. We can upload our courses to the cloud and send review invites to people. All the comments are collected in one place. Plus it has Screenr built into it. It’s been working really great for us and we decided to open it up to others.
James, the product your team has developed looks outstanding. It potentially addresses soooo many issues.
No offense meant, but I was hoping the Articulate folks would be first to the table with this, but I'm very glad someone came up with the solution! My company will be reaching out to you shortly to learn more.
IMHO this solution deserves it's own thread becaue it truly fills a gaping hole that many developers/teams face.
No offense taken. Although we (elearningenhanced.com) work primarily in Articulate productions we occasionally work in other's and wanted something that could handle those tools as well.
I'm glad you like it! We will be happy to answer any questions you have.
That looks fantastic James. I was just having the conversation yesterday about how we are going to get reviewers to give their feedback. I will sign up for a free trial and experiment with it, but just a question - is it very easy to use for non-techy people? My course is for pharmacy assistants (drug store staff I believe you would say in the US lol), so they may be quite computer savvy or may be fairly computer illiterate. I need to make the feedback process as simple as possible. Have you had much feedback on whether your site is easy to use?
Ease of use is one of the things that drove us to create this. We had a few clients that were struggling with Google Docs and they love ReviewMyElearning.com
Let me know if you have any questions or suggestions for us.
At the start of project we discuss what we are going to review, when and how we are going to review it. Reviewing a prototype and then getting a set of word documents that we need to interpret can take a long time.
Some times we've used a tool like screen-o-matic and grabbed the screen with annotations. It has helped the developer and also we've also had a commentary and seen the course in action.
My experience is that get the review process into prototype as soon as possible. I've worked on projects where we've worked on screens and scripts that just have worked for the client when they have become 'interactive' and we've all wasted a lot of time - regardless of the review process that we've used.
On earlier projects we've built a script editor where everyone would work collaboratively which has been a success. This was an internal app and sounds like http://www.reviewmyelearning.com/.
The main issue of marking and reviewing is that you are giving people 'flat' content and asking them to thinking 'interactively'. We try and work with a process of MVP - minimum viable product and do small prototyping. This makes the review process easier and allows people to review in the environment that the final delivery will be. It is not always possible - but where we can we try!
25 Replies
Sid,
At the very least, you might want to outline a few focus questions.This will serve a few purposes:
1. It will force you to reevaluate whether you've achieved your objectives before sending for review
2. It will give the stakeholders and SMEs a starting-point for discussion and feedback
...and...related to #2... **drumroll*
3. It will -- hopefully -- prevent them from just giving a random list of comments as long as your arm, which is what's likely to happen if you don't give them a point of (limited) focus. This is because, if you don't provide focus and tell them what you expect, they won't know what to give feedback on; consequently, the default position will be to either give feedback on everything in minute and painfully pontificating detail... or... feedback on nothing at all! Neither or which is helpful to you.
Hope that helps,
Leslie
May be send it via dropbox. You can also maintain an Excel sheet - Slide title and comments. Both you and your SMEs can update the Excel sheet with comments.
I agree with El. It's important to focus the feedback by reminding your reviewers what you're trying to accomplish.
Also, to Poornima's point, a spreadsheet can be used to track edits. We have SharePoint in our organization so I set up a tracking list for each course that goes for review and then reviewers can add comments, even attachments like screenshots or other documents.
In addition I always send a Word document version of the course so that reviewers have somewhere to make notes. I'll often send a link to the course out with the Word document attached to the email or available alongside the course in the LMS. I'll usually set up a meeting with the reviewers about a week after I send the course for review to gather their comments and go through the course with them. I find this especially helpful if this is the first time I'm working with that person.
I normally send a link to the course on DropBox, and a version saved as a Word Document, with "Directors Notes" and guidance, as already mentioned.
When we do the second QA, (notmally takes 2 cycles), I re-emphasize that this is THEIR course, for THEIR customers, and they now need to make a go:no go decision on whether I have imterpreted and implemented their requirements correctly. Very often an ID will be blamed for any failure - so make sure you close that particular door to them.
I usually try to get ONE set of documents.
Bruce
Bruce
Some very good advice there. Thanks.
Leslie
Thanks guys for all the good advice. I like th eidea of sending the word document and maintaining excel spreadsheet. I won't be able to use dropbox but I think I can use my team sharpoint site.
Well, I tried publishing my project to word, it published the project but without any images or texts in the slide. The slides are just blank.
Any ideas what may be going on?
I can't find it right now but Tom did a post on embedding a google doc to collect course comments/feedback. I really like that approach and you could also use any survey tool like Survey Gizmo/Survey Monkey. All of those automatically collect the feedback in a spreadsheet and allow you to make whatever fields you want (including drop-down lists for categories, etc.)
Instead of embedding you could also include a "Comments" link in the top menu or on every slide to do this too.
One of the benefits of this approach is that you can get comments from multiple people all at the same time and save it into a single document without any of the copy/paste combining of the data.
In the past I created a Sharepoint site that had the courses loaded as well as a spreadsheet for their comments. I also included a sample of the type of comment I was looking for. In my email out, I would tell them that we (the training team) had already done a spelling/grammar/ check and that the course adhered to our development principles such as colors, graphics and things. In the spreadsheet, I had a column for my comments back to the reviewer where I would indicate if I had made the edits, or why I chose not to.
Oh yes, another thing - I ALWAYS put a big text box somewhere on each slide with the slide number. I take it out for the final draft, but it makes commenting/reviewing MUCH easier for everyone.
Thanks all for these great ways fo sending material to the SME for review.
I am still not being able to publish the course to a word document and see teh images. As far as I understadn all the images in each slide should be in the word document when publised to word? But I just get blank pages, without the images or text in them
Re Mike's comment, I can't find it right now but Tom did a post on embedding a google doc to collect course comments/feedback
This thread includes a link to Tom's Screencast, to more comments, and to other helpful Google Docs/SL links
Hi Sid,
I attached a screen shot with a simple answer even tho' I'm assuming you already clicked "Show Layers" on properties when publishing to word. Each layer will print the images with each layer labled. The text shows under the base layer image.
Yes, I did all that. I am now getting an error. I think it has to do with having permission issues. I have to contact my IT admins to figure out what setting I need in word to make this owrk.
{Re-post from another thread}
This topic of the logisitics of SME/Stakeholder review seems to come up over and over.
In the advertising world there are lots of services/sites available that allow content to be reviewed, marked up, and the comments catalogued. Many even include review database and reviewer management features. I'm told this kind of remote review tool has become a "standard" amongst the best agencies that allow them to work remote artists/designers around the world.
Sadly there seems to be a hole for this kind when it comes to e-learning courses.
When I investigated this late last year, there were precious few options available to remotely review and comment on interactive Flash content (plenty for non-interactive Flash video). And only one I could find aimed directly at e-learning.
Seems to me there is a TREMENDOUS OPPORTUNITY for the folks at Articulate to leverage not only their authoring tool expertise but also LMS experience to offer this service...
Great thread here...has anyone ever used an 'project management' type tools for reviews, questions, tracking to items related to a course etc?
Seems like tools along the lines of Asana, Wrike, Freed Camp etc might work? Anyone had success or failure along these lines?
I've briefly tried Dispatch.io and so far it looks pretty good. It enables comments on documents, etc from Google Drive, Dropbox, etc
https://dispatch.io/faq
Have actually now moved in some cases to just FTP'ing content to my server, to get over the DropBox caching issues, but still use DB a lot,.
Bruce
I use my server all the time, FTP and send a link.
The problem with Dropbox is a performance one, i get comments about it being slow or not refreshing. Not good if it is the first time they are seeing it.
Hey Sid,
Not sure if you found a solution for this… We were having issues getting good reviews from SMEs, stakeholders, etc. We tried a lot of the options mentioned here, Google Doc, Excel, Dropbox etc. In the end we created our own solution. We can upload our courses to the cloud and send review invites to people. All the comments are collected in one place. Plus it has Screenr built into it. It’s been working really great for us and we decided to open it up to others.
You can check it out at ReviewMyElearning.com
Thanks
James
OUTSTANDING!!!!!!
James, the product your team has developed looks outstanding. It potentially addresses soooo many issues.
No offense meant, but I was hoping the Articulate folks would be first to the table with this, but I'm very glad someone came up with the solution! My company will be reaching out to you shortly to learn more.
IMHO this solution deserves it's own thread becaue it truly fills a gaping hole that many developers/teams face.
Thank you Bob,
No offense taken. Although we (elearningenhanced.com) work primarily in Articulate productions we occasionally work in other's and wanted something that could handle those tools as well.
I'm glad you like it! We will be happy to answer any questions you have.
That looks fantastic James. I was just having the conversation yesterday about how we are going to get reviewers to give their feedback. I will sign up for a free trial and experiment with it, but just a question - is it very easy to use for non-techy people? My course is for pharmacy assistants (drug store staff I believe you would say in the US lol), so they may be quite computer savvy or may be fairly computer illiterate. I need to make the feedback process as simple as possible. Have you had much feedback on whether your site is easy to use?
Thanks Fiona.
Ease of use is one of the things that drove us to create this. We had a few clients that were struggling with Google Docs and they love ReviewMyElearning.com
Let me know if you have any questions or suggestions for us.
James
Hi.
The review process!
At the start of project we discuss what we are going to review, when and how we are going to review it. Reviewing a prototype and then getting a set of word documents that we need to interpret can take a long time.
Some times we've used a tool like screen-o-matic and grabbed the screen with annotations. It has helped the developer and also we've also had a commentary and seen the course in action.
My experience is that get the review process into prototype as soon as possible. I've worked on projects where we've worked on screens and scripts that just have worked for the client when they have become 'interactive' and we've all wasted a lot of time - regardless of the review process that we've used.
On earlier projects we've built a script editor where everyone would work collaboratively which has been a success. This was an internal app and sounds like http://www.reviewmyelearning.com/.
The main issue of marking and reviewing is that you are giving people 'flat' content and asking them to thinking 'interactively'. We try and work with a process of MVP - minimum viable product and do small prototyping. This makes the review process easier and allows people to review in the environment that the final delivery will be. It is not always possible - but where we can we try!
Thought I would let everyone know that we just rolled out a bunch of new features!
http://www.reviewmyelearning.com/changelog
#3 is so true! You have to give them focus. And I love your reference to pontificating. I have experienced that in the past.
This discussion is closed. You can start a new discussion or contact Articulate Support.