I'd like to know how others are handling Kirkpatrick Level 1 and Level 2 evaluations using Articulate on an LMS.
My LMS vendor told me that I can only get reports back for one Articulate quiz. Apparently SCORM format will only track a single quiz. This aligns with the publishing options that I've seen in Articulate.
But, I want to use a robust evaluation strategy that includes both Level 1 and Level 2 evaluation for my elearning courses. So, I'd like to know how others are getting around the SCORM reporting limitations.
In most cases I would think people hyperlink to a 3rd party evaluation product, such as:
Vovici
Qualtrix
SharePoint
Survey Monkey,
etc...
As you noted, you can only report on one object which is usually reserved for the assessent. Even if you could, some LMS's don't allow you to see individual responses to questions asked so it wouldn't work anyway. I know with SkillSoft, you can only receive feedback in the format of:
Course Name
Last Access Date
First Access Date
Times Accessed
Duration in Course
COmpletion Date
Pretest score
High score
Current Score
Now, one piece of advice if you decide to go the 3rd-party evaluation route. Evaluation hyperlinks change for many different reasons sometimes. Therefore, what I've done is hyperlink to a landing page that I can easily edit. That way, if the hyperlink to a course evaluation changes, I never have to upload the entire course again.
We get around this only asking some basic review questions in our courses and then using the built in quiz/test features in our LMS. Our user group likes to take courses together under one persons account - it is quicker and easier for them, so we just have to adapt - and we needed a way to email out tests for them to take manually, and going through the LMS we were able to do this.
I am working on this problem right now. I think our finals may go to Survey Monkey, but for quizzes, I am trying to design out a system using stand-alone Quizmaker and Outlook. You can have each student email their quiz to a dedicated mailbox in Outlook.
Quizmaker will send a comma delimited text file for each student. There first question should be (What is your ID?).
So that works fine, but before I implement this, I need to find a way to score these quickly, maybe as a batch file.
Any ideas for easy implementation?
Finally, why am I doing this? Are new hires have volumes of information to learn. They can test out with high scores but that doesn't mean it is humanly possible for them to remember all that content. So, I would like to institute some "interval repetition" quizzes through Outlook that would assist them after moving on from the classes.
@ Deborah, check out this tool called Mindsetter. You can have your articulate courses, and then set up an automated 'interval repetition' system through Mindsetter. You can also attach some mini chunks of your content to get automatically emailed out.
Eric--My LMS does have authoring capability which would allow me to build quizzes and surveys. But, the LMS tool doesn't have the item type versatility that Quizkmaker does. Also, our LMS course author tool is essentially an HTML editor. Most of the people on my team don't have the skills to use it well, so I'm trying to avoid that. Wish it did have a better interface because that's what we would choose to do.
Our current training is running off our SharePoint site. We've got our survey results being mailed to our corporate university mailbox. Works great. We change the survey title to match the course and that shows up in the re: line of the email. You probably won't need ID as your first question, because you'll see the sender's email address.
One note: make sure you have complete instructions on the "results" page...to click to email and click "allow" for the student's email program to send. Otherwise your learners might get confused.
I'm not doing batch processing. I set up an Excel spreadsheet with pull down menus and am entering responses by hand. The body of the email contains the results so it's easy to do. I'd think you'd need Articulate to attach a .csv file to the email instead of dropping it as text into the body of the email in order to batch process. I'm sure there must be a way to do that, but it will probably involve rewriting the published .xml files somehow. Anyone got ideas on this?
Thank you for your responses. Mindsetter seems pretty slick and it definitely has most of the features I want, but for the number of people we run through our classes monthly, the cost is more than our unit can justify. I really like it though. =0)
I will look at Excel. Quizmaker will send a .csv file, Outlook will do the mind trigger step for me. I will look at Anki and Mnemosyne too. Let you know if I find a good combination!
We use Quizmaker to track Level 2, and thankfully, our LMS includes post-training evaluation surveys (Level 1) with automatic sending, reminding, and reporting. I get "Metrics That Matter" reports monthly for each course I "own" as well as one compilation report for all courses I own. There are things I really don't like about our LMS (I *think* it's running on PeopleSoft or whatever company they are now), but this particular feature is quite handy.
Maybe check with your LMS to see if this capability is there?
Rebekah--Our LMS will allow me to add a survey, but I'd have to build a "course" with their authoring tool. I can import the SCORM Articulate file, but I have to build a static HTML page and then put the course in that page. The LMS editor is like looking at a blank Word document. In order to get it to do what I want, I'd have to be able to write code including some JavaScript. I just don't have those skills. And, the LMS quiz tool only offers multiple choice or t/f unlike QuizMaker. The survey tool only offers Likert scaling. Long and short...it's just not the right tool for someone who isn't a programmer and wants versatility. :(
I'm still working with the LMS help desk to obtain reports from SCORM quizzes. They didn't understand why I would want to see responses to individual questions or items. Apparently, the tool was designed by IT folks with little or no input from training professionals. When I started talking to them about Level 1 and Level 2 instruments, they didn't know what I meant. I had to explain it to them, then explain how they differ and are used, and then explain why I need individual responses. I'm considering charging them a consulting fee!
Yes, I would love to have an LMS do the work for me. We are tied to work-force management and need to deliver courses within a few days of completion. The operations unit cannot guarantee a consistent date before two weeks from request... so the turn around time just isn't fast enough. We are presenting off of a Sharepoint site, also. thanks for the idea though.
11 Replies
In most cases I would think people hyperlink to a 3rd party evaluation product, such as:
As you noted, you can only report on one object which is usually reserved for the assessent. Even if you could, some LMS's don't allow you to see individual responses to questions asked so it wouldn't work anyway. I know with SkillSoft, you can only receive feedback in the format of:
Now, one piece of advice if you decide to go the 3rd-party evaluation route. Evaluation hyperlinks change for many different reasons sometimes. Therefore, what I've done is hyperlink to a landing page that I can easily edit. That way, if the hyperlink to a course evaluation changes, I never have to upload the entire course again.
Has anyone tried setting the LMS to track the Level 2 evaluation, and setting the Level 1 evaluation to email the results?
Hey Rebecca,
We get around this only asking some basic review questions in our courses and then using the built in quiz/test features in our LMS. Our user group likes to take courses together under one persons account - it is quicker and easier for them, so we just have to adapt - and we needed a way to email out tests for them to take manually, and going through the LMS we were able to do this.
Hi,
I am working on this problem right now. I think our finals may go to Survey Monkey, but for quizzes, I am trying to design out a system using stand-alone Quizmaker and Outlook. You can have each student email their quiz to a dedicated mailbox in Outlook.
Quizmaker will send a comma delimited text file for each student. There first question should be (What is your ID?).
So that works fine, but before I implement this, I need to find a way to score these quickly, maybe as a batch file.
Any ideas for easy implementation?
Finally, why am I doing this? Are new hires have volumes of information to learn. They can test out with high scores but that doesn't mean it is humanly possible for them to remember all that content. So, I would like to institute some "interval repetition" quizzes through Outlook that would assist them after moving on from the classes.
@ Deborah, check out this tool called Mindsetter. You can have your articulate courses, and then set up an automated 'interval repetition' system through Mindsetter. You can also attach some mini chunks of your content to get automatically emailed out.
Eric--My LMS does have authoring capability which would allow me to build quizzes and surveys. But, the LMS tool doesn't have the item type versatility that Quizkmaker does. Also, our LMS course author tool is essentially an HTML editor. Most of the people on my team don't have the skills to use it well, so I'm trying to avoid that. Wish it did have a better interface because that's what we would choose to do.
Deborah--
Our current training is running off our SharePoint site. We've got our survey results being mailed to our corporate university mailbox. Works great. We change the survey title to match the course and that shows up in the re: line of the email. You probably won't need ID as your first question, because you'll see the sender's email address.
One note: make sure you have complete instructions on the "results" page...to click to email and click "allow" for the student's email program to send. Otherwise your learners might get confused.
I'm not doing batch processing. I set up an Excel spreadsheet with pull down menus and am entering responses by hand. The body of the email contains the results so it's easy to do. I'd think you'd need Articulate to attach a .csv file to the email instead of dropping it as text into the body of the email in order to batch process. I'm sure there must be a way to do that, but it will probably involve rewriting the published .xml files somehow. Anyone got ideas on this?
Eric and Rebecca,
Thank you for your responses. Mindsetter seems pretty slick and it definitely has most of the features I want, but for the number of people we run through our classes monthly, the cost is more than our unit can justify. I really like it though. =0)
I will look at Excel. Quizmaker will send a .csv file, Outlook will do the mind trigger step for me. I will look at Anki and Mnemosyne too. Let you know if I find a good combination!
Thank you so much for responding =0)
We use Quizmaker to track Level 2, and thankfully, our LMS includes post-training evaluation surveys (Level 1) with automatic sending, reminding, and reporting. I get "Metrics That Matter" reports monthly for each course I "own" as well as one compilation report for all courses I own. There are things I really don't like about our LMS (I *think* it's running on PeopleSoft or whatever company they are now), but this particular feature is quite handy.
Maybe check with your LMS to see if this capability is there?
Rebekah--Our LMS will allow me to add a survey, but I'd have to build a "course" with their authoring tool. I can import the SCORM Articulate file, but I have to build a static HTML page and then put the course in that page. The LMS editor is like looking at a blank Word document. In order to get it to do what I want, I'd have to be able to write code including some JavaScript. I just don't have those skills. And, the LMS quiz tool only offers multiple choice or t/f unlike QuizMaker. The survey tool only offers Likert scaling. Long and short...it's just not the right tool for someone who isn't a programmer and wants versatility. :(
I'm still working with the LMS help desk to obtain reports from SCORM quizzes. They didn't understand why I would want to see responses to individual questions or items. Apparently, the tool was designed by IT folks with little or no input from training professionals. When I started talking to them about Level 1 and Level 2 instruments, they didn't know what I meant. I had to explain it to them, then explain how they differ and are used, and then explain why I need individual responses. I'm considering charging them a consulting fee!
Yes, I would love to have an LMS do the work for me. We are tied to work-force management and need to deliver courses within a few days of completion. The operations unit cannot guarantee a consistent date before two weeks from request... so the turn around time just isn't fast enough. We are presenting off of a Sharepoint site, also. thanks for the idea though.
This discussion is closed. You can start a new discussion or contact Articulate Support.