Going from paper review to online

Just thinking out loud here, but could you present the learners with mock policies, just as you do in the in-person training? You could put these in Storyline's scrolling panels if necessary to allow learners to scroll through them if they are long.

Put hotspots over phrases that are correct as-is and over phrases that are incorrect. Have learners click the phrases they think need rewriting.

Whenever a learner clicks a phrase on the document (actually, the learner will be clicking the hotspot you put over that part of the document), pop up a layer that asks learners to pick one of three possible rewordings.

In this way, your learners can kind of virtually edit and rewrite the documents in your course.

Ray, thank you for this idea. I will see what I can work out and let folks know!

My first thought corresponds with Ray's suggestion and is probably the easiest to implement. It is easy to record whether all hotspots were selected and to provide immediate feedback for each hotspot. You can even provide feedback if the reviewer selects content which is not incorrect. However, this method does not obviously allow the reviewer to explain why they think the content is incorrect or to provide alternative wording.

An alternative might be to use a document editor to number each line of the policy, place it in a scrolling panel (as above) and ask the reviewer to record (in a text entry field) the relevant line number(s) where deviation with policy occurs with an explanation why or alternative wording. If screen size/space allows, you could have the text entry field next to the policy document (both in scrolling panels). Depending on the size of the reviewer text responses, the reviewer should be able to print and email the response to an instructor for comment (using methods provided elsewhere in ELH) or you could use Articulate Review to enable an instructor to comment/provide feedback.

It seems that this kind of learning activity really requires someone checking the learner submissions for it to be fully effective. Will there be someone doing this or are you attempting to automate completely?

If not, I wonder whether you offer the policy documents as resources for the learner to download. You could then set up an activity where you ask the learner to mark up the document with any changes before moving ahead to compare notes with the 'expert'. Here you can share best practice. You might be worried that the learner may just skip ahead without completing the offline activity but you could possibly validate this with an assessment at the end to at least check that the best practice has been absorbed.

Hello folks,

Thanks for all your ideas. Boy howdy, this is a puzzler. Rise is pretty darned neato torpedo, though.

I already have the learners look at a properly written policy document that has clickable markers placed at points that are often troublesome issues. Click the marker, read the popup content, go to the next.

Then I have them look at scenarios that may, or may not, have issues. They either click flashcards for commentary, fill in the blank to complete sentences, or answer multiple choice questions.

My client's desire, thus my desire(!), is to emulate how policy review and revision is done currently in real time, e.g. the learners gets a printed erroneous policy document, marks it up, then an instructor reviews the markups and discusses with them. An important issue is to not only revise the incorrect content, but avoid messing with the stuff that is ok as is. But, do all this  automatically. The real trick seems to be automating the review part. 

So, here is what I will propose to my client: after going through all that above activities, have the learner download an editable Word version of the erroneous policy document, mark it up using Track Changes, then email that to their manager/mentor/whatever for follow-up.

Seems to be a bit of a step back from our automation desires, but unless anyone has any better ideas, I think I will have to go with that.

To follow up, what I sent to my client is this:

1. If we use an approach where the “erroneous” document has every paragraph as a separate quiz question, we could go with essay style revisions. However, given the variety of correct input possible, unless we could settle on required keywords in each response, automated review would be futile. The quiz could not assess if the response is correct.

2. We use the same divided paragraph style and present the learner with 3 multiple choice selections of proposed rewording on each paragraph. They would be required to select the best rewording, plus have a choice to say the paragraph is fine as is, of course. Each paragraph would have a single correct response, thus we could grade the quiz and obtain the learner’s answers for review if needed.

The down sides of doing that would be restricting the learners’ responses to those we give them, not to mention the work involved in devising those responses on our parts, but also this would break up the document into somewhat disconnected component parts.

3. Another idea is to have the “erroneous” document as a downloadable and editable Word .docx file, have the learner mark it up using Track Changes, then email that to their manager/mentor for follow-up. This would keep the policy document as a whole entity and allow the learner a free hand to write what they want to. Of course, that would mean hand-work on the part of the manager, but also would allow the manager the opportunity to analyze the learner’s thinking, writing, and ability to express thoughts well, then to mentor the learner.

4. Finally, we could blend some. Use the multiple choice approach but add a 5th selection – “I would reword this differently.” This selection will always be correct; however, it would not stand out in the grading. The learner could get a 100%, including some “I would reword…” responses. After the quiz is submitted and graded, the manager would still need to access the quiz and give the answers a once over; if the learner would reword differently, that would be an opportunity to discuss things.

I could go on, but I won't. I shall let folks know the outcome.