Hello all. Anyone know of a good way to collect feedback from users on feedback on deployed courses? I am sending my course to my client to review and they want to know the best way to provide feedback to me outside of screen captures and Word documents?
Nothing wrong with Word documents, especially now you have the Translation Export / Import functionality.
Send an "Export" out --> get SMEs to review and come up with agree text via ONE of them --> they create the new words on the translation document --> you Import it back in.
Completed course text, ready for animation and images.
The short answer.... there are not a lot of good options beyond what you've listed. The only product/service I know of is from RapidIntake and it's called Review.
The long answer... (reposted from other posts)
This topic of the logisitics of SME/Stakeholder review seems to come up over and over.
In the advertising world there are lots of services/sites available that allow content to be reviewed, marked up, and the comments catalogued. Many even include review database and reviewer management features. I'm told this kind of remote review tool has become a "standard" amongst the best agencies that allow them to work remote artists/designers around the world.
Sadly there seems to be a hole for this kind when it comes to e-learning courses.
When I investigated this late last year, there were precious few options available to remotely review and comment on interactive Flash content (plenty for non-interactive Flash video). And only one I could find aimed directly at e-learning.
Seems to me there is a TREMENDOUS OPPORTUNITY for the folks at Articulate to leverage not only their authoring tool expertise but also LMS experience to offer this service...
Thanks all for your input. I do have approved scripts in Word that the client approved that I used to build the course. So if they make changes solely to content, they can mark up those scripts. But regarding general comments on the course in general, I guess I will have to reply on screen captures and mark up.
I will look at Review as well, thanks Bob.
I will make another feature request to the folks! Thanks again for you responses :0)
What I don't get is the obsession with "new tools". Why can't we just use the ones we have more effectively.
It's not beyond the whit of man to say:
1> People - here's my course. Click on the link to review, and write down your thoughts.
2> Get on a conference call, and discuss each point (yes....real communication. I know it's scary but......)
3> Nominate a "spokesperson". Granted you may need another sub-committee and process to do that....
4> Get that spokesperson to make the decisions for you
5> Update the Word document.
I appreciate that I am "old skool", but I've run dozens Global projects on pretty much that strategy for many years. It's usually quickest, and people actually like the debate that ensures. If you really want to get all techno, you can EVEN use different colors or fonts for each person's thoughts.......
If you have set the design procedure up anyway, there should not be an interface of more than 2 or 3 people with you, the Instructional Designer anyway.
Maybe, (just maybe...) there's "...a hole..." because it is over-engineering something that's really quite simple if you understand communication, motivation and project management?
Bruce
PS - "...Old age, experience and cunning will always win over youthful misplaced enthusiasm..."
PPS - you can always set up a blog or Wordpress page or something for ongoing reviews....
Agree with Bob. There's a great opportunity for someone to field an easy to use collaborative review / reporting service. There are some challenges to using something like this for Articulate Presenter and Storyline since they're contained within a single page and the unique id's that would make matching content to discrepancies aren't exposed.
I built something years ago (a decade ago) that captured discrepancies in an online database (visible to all reviewers) and provided reports. David Anderson posted a presenter example that showed a Wiki in a Web Object for this purpose.
Lots of options but it sure would be great to have something built for the purpose. I'm betting there's a market for a solid review tool. Rapid Intake's Review is good but it's expensive and I think something simpler and cheaper could really sell well.
Dunno Bruce. In some cases we have between 5 and 15 stakeholders for final review. Add to that pilot users and the case for a simple / single stream to correlate all the input and track resolution becomes a bit more clear
For smaller focused things, yeah. I use Excel and Word docs. Works well in parties of three. Works really well. But there are times when it gets larger than that I'd really like to drive some discussion around points but don't want to be in the middle. A simple thread attached to a page would let that happen without me getting in the middle:)
Perhaps this comes from my personal perspective....
As a freelancer, I work quite hard at selling concepts, workflows and decision structures that suit me and the way I need to work.
I always try and retain a 1:1 relationship with the person that I deal with, so I never have to deal with huge teams.
Currently building a course for many hundreds of thousands of potential learners, I will still only deal with 1 person that will deal with 2 others, and yes, we WILL assess feedback.
Even when I worked at Oracle I ran things this way - I just find it easier to do, even when there is a whole group of stakeholders, I do not view it as necessarily my role to get caught up in their reviews etc.
They can do that without me, I do not add a lot to the process, so I engineer the situations to run in another way.
In addition to Steve's point above about multiple traditional stakeholders, another thing to keep in mind is that(like it or not) regulatory training is being mandated more and more often.
Accordingly, there is a growing need to have multiple departments as well as government auditors/regulators review courseware to ensure that it satisfies requirements... including those silly length requirements. I can tell you this is a bitter reality in the financial services/banking world where there are often 2 or 3 agencies that have a "stake" in the training as well as internal departments such as governance, compliance, legal, etc. And I'm told the healthcare industry is another example where this sort of thing is rampant.
WIth the growth in regulatory training, the need for a collaborative review tool/service only increases.
I guess I'm just in a minority with using a well-tried and successful system then.... :(
Bruce
Whatever works for you is golden. No right or wrong here.
I know a local company here that was trying to reach out to Articulate with their review tool. Think that was a couple of years back. If memory serves, don't believe Articulate was interested back then . . .
I have had success using a google-doc based spreadsheet that all the stakeholders can access and make their comments in, on a screen by screen basis. You can customize it pretty much any way you want, it frees the process from a round-robin format and/or associated version control issues of an emailed spreadsheet and allows all the users to review and post comments on their own time. A conference call to clarify comments and resolve conflicting comments is the next step. Color coding and a column for dates and notes fields can be used to keep track of changes, and of course, the entire history of changes are kept in one place for all to refer to.
I thought I'd mention that in the future (IMHO) people taking the training really need a way to make comments/add suggestions to the course creator/person responsible for updates. I would personally argue they are are just as if not more important than stakeholders, content experts, etc.
Not to state the obvious, but with the complexity of layering and non linear navigation, one problem for someone who didn't create the training is to even accurately describe what's wrong. (They normally have no idea what layer is showing/supposed to be showing, what "slide" they are on, what trigger was activated, etc.)
If you really wanted a revolutionary review tool, there would be an easy way (if enabled) for any user to click pause, click on a submit/comment button in the player and the results would be transmitted via email or stored, etc. (along with any time code/indication of what slide/layer was showing/maybe in my dreams an automatic screenshot attachment) to the person in charge of updating the course.
I've never tried Review Link and that may be how it works, but I think the word "review" might need to be extended to anyone viewing and there's no real easy way to do that now in Storyline.
That's a great point, Todd. A few years back we built in a feedback form that provided a set of choices and a big open field. Behind the scenes, it captured the location ID within the course as well as some basic client info (browser, IP, etc.) and submitted any captured reports to a database. This was really useful in the beginning of the roll-out. After awhile, most of the feedback we got was complaining about the nature of the training, not the content or execution. Still a pretty good exercise in evaluating both the solution and the audience response over time.
In my Storyline files, I've added a storyboard ID. I even have extensions of this variable that change for a few layer additions. The original intent was to add in a capture utility, but in the short run I added a display of this tag in a visible place during review. That way reviewers could tell me *exactly* where a problem was.
The base storyboard for a segment looks something like this: s1c1. For a layer or layer set, I may extend this with s1c1a, s1c1b, or s1c1_activity... Whatever I need to get an absolute reference. It's proven handy. It's also not difficult to add and modify these numbers from the Story View. 5 to 10 minutes during setup to get the base storyboard id's added once the structure is put together.
@steve Very interesting. My consistency with naming conventions has know to be lacking so I'd probably confuse myself.
I was viewing a webinar earlier today and interestingly enough the video player at MIT Tech TV was mentioned because it allows inline comments. I don't know how long it's been available, but it sounds similar to what I was describing. (At least for video)
In-line commenting / video chaptering: Allows users to comment on a specific point in time on the video. Also creates an associated link below in the page comment section with a link that will jump to that spot in the video where the comment was made.
On the surface, this functionality appears similar to the services mentioned above that Creative Agencies have been using for a while. Namely, the abilitity to comment and even "white-board" drawings on video clips for review. The challenge has been that those services play nice with regular Flash video.... not interactive Flash like games or elearning.
I can't help but think Articulate might be the pefect company to come out with something that only one other vendor seems to offer. With Articulate's reach and reputation, they could corner the market for collaborative e-learning review and more.
Hey all, have found a solution for my review tool problem! I am working with Allen Interactionson a few projects. Together we figured out how to bring in a Flash app to Storyline that adds a widget to each page of the course. The widget (green icon that users click) pops open a review window where users can add their feedback. Then all the data gets collected and can be managed real time in a web site that our team and client can review those comments and mitigate them. Pretty slick! See capture.
Not sure about that, we pulled up some old and new technology and got it to work. I am just happy we got it to work in Storyline, saves me a lot of hassle! Just thought I would share.
Interesting. It's a neat idea. Worth a bit of tickling this weekend to see if I can get something integrated with Trac. Of all the bug trackers I've loved before, Trac is the one I loved the most
Here's a trac report example. Pretty powerful (and overly complex in some ways) but I think it can be leaned out to just the ticket reports and comments.
So I take it the final answer here is to use a word doc or spreadsheet of some sort and there is not a review tool available - yet. I hope Articulate was reading this discussion. I've worked with a company which had a review tool for their CBTs and it was pretty awesome. You could review each screen, log comments and tag them based on your role (ISD or Graphic Artist) plus who would be responsibel for correcting the issue. So if it was a functionality issue it would be assigned to the designer, etc. Then reports could be run. One other thing that I've noticed that Storyline is in need of is some sort of numbering for the pages. I realize that with truly branching scenarios and such you may not want to have page numbers, but it would be nice to have that option to add it in if you're just creating a quick fairly linear module. It makes it easier to find the page that needs editing or is having functionality issues. Just my two cents!
Not sure if you found a solution for this… We were having these same issues, so we created our own solution. We can upload our courses to the cloud and send review invites to people. It tracks what slide they are on and they can tag their comments. All the comments are collected in one place. Plus it has Screenr built into it. It’s been working really great for us and we decided to open it up to others.
65 Replies
Hi Jeff.
Nothing wrong with Word documents, especially now you have the Translation Export / Import functionality.
Send an "Export" out --> get SMEs to review and come up with agree text via ONE of them --> they create the new words on the translation document --> you Import it back in.
Completed course text, ready for animation and images.
Just a thought.
Bruce
Hi Jeff,
The short answer.... there are not a lot of good options beyond what you've listed. The only product/service I know of is from RapidIntake and it's called Review.
The long answer... (reposted from other posts)
This topic of the logisitics of SME/Stakeholder review seems to come up over and over.
In the advertising world there are lots of services/sites available that allow content to be reviewed, marked up, and the comments catalogued. Many even include review database and reviewer management features. I'm told this kind of remote review tool has become a "standard" amongst the best agencies that allow them to work remote artists/designers around the world.
Sadly there seems to be a hole for this kind when it comes to e-learning courses.
When I investigated this late last year, there were precious few options available to remotely review and comment on interactive Flash content (plenty for non-interactive Flash video). And only one I could find aimed directly at e-learning.
Seems to me there is a TREMENDOUS OPPORTUNITY for the folks at Articulate to leverage not only their authoring tool expertise but also LMS experience to offer this service...
Hope this helps,
Bob
Thanks all for your input. I do have approved scripts in Word that the client approved that I used to build the course. So if they make changes solely to content, they can mark up those scripts. But regarding general comments on the course in general, I guess I will have to reply on screen captures and mark up.
I will look at Review as well, thanks Bob.
I will make another feature request to the folks! Thanks again for you responses :0)
What I don't get is the obsession with "new tools". Why can't we just use the ones we have more effectively.
It's not beyond the whit of man to say:
1> People - here's my course. Click on the link to review, and write down your thoughts.
2> Get on a conference call, and discuss each point (yes....real communication. I know it's scary but......)
3> Nominate a "spokesperson". Granted you may need another sub-committee and process to do that....
4> Get that spokesperson to make the decisions for you
5> Update the Word document.
I appreciate that I am "old skool", but I've run dozens Global projects on pretty much that strategy for many years. It's usually quickest, and people actually like the debate that ensures. If you really want to get all techno, you can EVEN use different colors or fonts for each person's thoughts.......
If you have set the design procedure up anyway, there should not be an interface of more than 2 or 3 people with you, the Instructional Designer anyway.
Maybe, (just maybe...) there's "...a hole..." because it is over-engineering something that's really quite simple if you understand communication, motivation and project management?
Bruce
PS - "...Old age, experience and cunning will always win over youthful misplaced enthusiasm..."
PPS - you can always set up a blog or Wordpress page or something for ongoing reviews....
Agree with Bob. There's a great opportunity for someone to field an easy to use collaborative review / reporting service. There are some challenges to using something like this for Articulate Presenter and Storyline since they're contained within a single page and the unique id's that would make matching content to discrepancies aren't exposed.
I built something years ago (a decade ago) that captured discrepancies in an online database (visible to all reviewers) and provided reports. David Anderson posted a presenter example that showed a Wiki in a Web Object for this purpose.
Lots of options but it sure would be great to have something built for the purpose. I'm betting there's a market for a solid review tool. Rapid Intake's Review is good but it's expensive and I think something simpler and cheaper could really sell well.
Dunno Bruce. In some cases we have between 5 and 15 stakeholders for final review. Add to that pilot users and the case for a simple / single stream to correlate all the input and track resolution becomes a bit more clear
For smaller focused things, yeah. I use Excel and Word docs. Works well in parties of three. Works really well. But there are times when it gets larger than that I'd really like to drive some discussion around points but don't want to be in the middle. A simple thread attached to a page would let that happen without me getting in the middle:)
I hear you Steve....
Perhaps this comes from my personal perspective....
As a freelancer, I work quite hard at selling concepts, workflows and decision structures that suit me and the way I need to work.
I always try and retain a 1:1 relationship with the person that I deal with, so I never have to deal with huge teams.
Currently building a course for many hundreds of thousands of potential learners, I will still only deal with 1 person that will deal with 2 others, and yes, we WILL assess feedback.
Even when I worked at Oracle I ran things this way - I just find it easier to do, even when there is a whole group of stakeholders, I do not view it as necessarily my role to get caught up in their reviews etc.
They can do that without me, I do not add a lot to the process, so I engineer the situations to run in another way.
Bruce
As for review tools, I was going to try what Tom describes here in my next project: https://player.vimeo.com/video/204930805
I like having something do the collecting for me. Any try this?
In addition to Steve's point above about multiple traditional stakeholders, another thing to keep in mind is that(like it or not) regulatory training is being mandated more and more often.
Accordingly, there is a growing need to have multiple departments as well as government auditors/regulators review courseware to ensure that it satisfies requirements... including those silly length requirements. I can tell you this is a bitter reality in the financial services/banking world where there are often 2 or 3 agencies that have a "stake" in the training as well as internal departments such as governance, compliance, legal, etc. And I'm told the healthcare industry is another example where this sort of thing is rampant.
WIth the growth in regulatory training, the need for a collaborative review tool/service only increases.
Bob
I guess I'm just in a minority with using a well-tried and successful system then.... :(
Bruce
Whatever works for you is golden.
I know a local company here that was trying to reach out to Articulate with their review tool. Think that was a couple of years back. If memory serves, don't believe Articulate was interested back then . . .
I have had success using a google-doc based spreadsheet that all the stakeholders can access and make their comments in, on a screen by screen basis. You can customize it pretty much any way you want, it frees the process from a round-robin format and/or associated version control issues of an emailed spreadsheet and allows all the users to review and post comments on their own time. A conference call to clarify comments and resolve conflicting comments is the next step. Color coding and a column for dates and notes fields can be used to keep track of changes, and of course, the entire history of changes are kept in one place for all to refer to.
Dare I speak of the competition, but Lectora had one thing right with Review link. Maybe someday Articulate will have a similar product!
I thought I'd mention that in the future (IMHO) people taking the training really need a way to make comments/add suggestions to the course creator/person responsible for updates. I would personally argue they are are just as if not more important than stakeholders, content experts, etc.
Not to state the obvious, but with the complexity of layering and non linear navigation, one problem for someone who didn't create the training is to even accurately describe what's wrong. (They normally have no idea what layer is showing/supposed to be showing, what "slide" they are on, what trigger was activated, etc.)
If you really wanted a revolutionary review tool, there would be an easy way (if enabled) for any user to click pause, click on a submit/comment button in the player and the results would be transmitted via email or stored, etc. (along with any time code/indication of what slide/layer was showing/maybe in my dreams an automatic screenshot attachment) to the person in charge of updating the course.
I've never tried Review Link and that may be how it works, but I think the word "review" might need to be extended to anyone viewing and there's no real easy way to do that now in Storyline.
Just my two cents.
Todd
That's a great point, Todd. A few years back we built in a feedback form that provided a set of choices and a big open field. Behind the scenes, it captured the location ID within the course as well as some basic client info (browser, IP, etc.) and submitted any captured reports to a database. This was really useful in the beginning of the roll-out. After awhile, most of the feedback we got was complaining about the nature of the training, not the content or execution. Still a pretty good exercise in evaluating both the solution and the audience response over time.
In my Storyline files, I've added a storyboard ID. I even have extensions of this variable that change for a few layer additions. The original intent was to add in a capture utility, but in the short run I added a display of this tag in a visible place during review. That way reviewers could tell me *exactly* where a problem was.
The base storyboard for a segment looks something like this: s1c1. For a layer or layer set, I may extend this with s1c1a, s1c1b, or s1c1_activity... Whatever I need to get an absolute reference. It's proven handy. It's also not difficult to add and modify these numbers from the Story View. 5 to 10 minutes during setup to get the base storyboard id's added once the structure is put together.
@steve Very interesting. My consistency with naming conventions has know to be lacking so I'd probably confuse myself.
I was viewing a webinar earlier today and interestingly enough the video player at MIT Tech TV was mentioned because it allows inline comments. I don't know how long it's been available, but it sounds similar to what I was describing. (At least for video)
In-line commenting / video chaptering: Allows users to comment on a specific point in time on the video. Also creates an associated link below in the page comment section with a link that will jump to that spot in the video where the comment was made.
Looks a little like viddler. I think this'd be a great feature addition. Adding a social element to something that's normally an isolated affair.
Hi Todd and Steve,
On the surface, this functionality appears similar to the services mentioned above that Creative Agencies have been using for a while. Namely, the abilitity to comment and even "white-board" drawings on video clips for review. The challenge has been that those services play nice with regular Flash video.... not interactive Flash like games or elearning.
I can't help but think Articulate might be the pefect company to come out with something that only one other vendor seems to offer. With Articulate's reach and reputation, they could corner the market for collaborative e-learning review and more.
Bob
Hey all, have found a solution for my review tool problem! I am working with Allen Interactionson a few projects. Together we figured out how to bring in a Flash app to Storyline that adds a widget to each page of the course. The widget (green icon that users click) pops open a review window where users can add their feedback. Then all the data gets collected and can be managed real time in a web site that our team and client can review those comments and mitigate them. Pretty slick! See capture.
Outstanding, Jeff.
Is Allen Interactions going to market this thing? (read as: "gimme gimme gimme!")
Bob
Not sure about that, we pulled up some old and new technology and got it to work. I am just happy we got it to work in Storyline, saves me a lot of hassle! Just thought I would share.
Interesting. It's a neat idea. Worth a bit of tickling this weekend to see if I can get something integrated with Trac. Of all the bug trackers I've loved before, Trac is the one I loved the most
Here's a trac report example. Pretty powerful (and overly complex in some ways) but I think it can be leaned out to just the ticket reports and comments.
http://trac.edgewall.org/report/1
So I take it the final answer here is to use a word doc or spreadsheet of some sort and there is not a review tool available - yet. I hope Articulate was reading this discussion. I've worked with a company which had a review tool for their CBTs and it was pretty awesome. You could review each screen, log comments and tag them based on your role (ISD or Graphic Artist) plus who would be responsibel for correcting the issue. So if it was a functionality issue it would be assigned to the designer, etc. Then reports could be run. One other thing that I've noticed that Storyline is in need of is some sort of numbering for the pages. I realize that with truly branching scenarios and such you may not want to have page numbers, but it would be nice to have that option to add it in if you're just creating a quick fairly linear module. It makes it easier to find the page that needs editing or is having functionality issues. Just my two cents!
Hey all,
Not sure if you found a solution for this… We were having these same issues, so we created our own solution. We can upload our courses to the cloud and send review invites to people. It tracks what slide they are on and they can tag their comments. All the comments are collected in one place. Plus it has Screenr built into it. It’s been working really great for us and we decided to open it up to others.
You can check it out at ReviewMyElearning.com
Thanks
James
This discussion is closed. You can start a new discussion or contact Articulate Support.