Everything we know about Cornerstone on Demand and Storyline!

Sep 04, 2013

Jason Chan
By Jason Chan

I thought I would start a thread about Cornerstone on Demand so there's a central place for people to read and add to. There seems to be a dearth of information about how CSOD and Storyline interact with each other as well as the unresolved issues surrounding it. I've spent many hours scouring the web and speaking with Cornerstone trying to find a solution. Hopefully this thread will serve as a resource for others going forward.

On a side note, I love Articulate Storyline. I started with Adobe Captivate with limited success. It was a very steep learning curve, but I was able to get some basic functionality out of it. I tested out Storyline with it's 30 day trial, and I will say that time-to-effectiveness was drastically reduced. It really IS like using powerpoint. Granted there is less complex functionality and interactions available than Captivate, but so far it's given me everything I've needed to use. The benefit is that the learning curve is much less steep too. Think of it as the difference between iMovie and Final Cut Pro. The latter can do some crazy effects, but the former gives you 90% of what you need and can get you comfortable in 1/4 of the time.

It's nice to have a properly-working content creation tool when you're in an LMS like Cornerstone. I have enough trouble in CSOD as it is.

So some of the nuggets of wisdom that I've found are :

  • Cornerstone does not track SCORM 1.2 content properly. You cannot pull a lot of the reports if they are not in SCORM 2004 format. You don't have the option to select that course in the reports.
  • If you export into SCORM 2004, there will be an extra section on the left with text links to the module. This takes up 30% of the screen, and on smaller laptop screens may crowd out the module window. Cornerstone says this is a Storyline exporting settings issue. I couldn't find any settings that shows/hides that section. The workaround is to resize the module to the screen, which doesn't get rid of the section, but at least fits the module.
  • When reporting, you can get the completion status for a course based on quiz result or last slide viewed. You can also get a final grade for course. Storyline passes the results information for each question (including short answer) but you cannot pull a list of the answers for an individual. Therefore, it may not be advisable to administer any testing where you'll need to review individual answers
  • You can get a list of aggregate answers for a module. e.g. I can see how 100 employees answered this multiple choice question. This is helpful to determine if learners are absorbing the course information correctly.
  • Web objects (such as websites, or links to sharepoint documents) work in modules if your IT has added csod.com to the trusted sites
  • Popup windows that open up the articulate module in CSOD will work if you add csod.com to the allowed list for popup blockers. Your IT dept can also add this to everyone's computer. Otherwise when learners click the "launch" button, the module does not show up.

I'll post more as I think or find them, but feel free to add to this list!

906 Replies
Chris Undery
almost 6 years ago05/19/18 at 7:33 am (UTC)

I'm trying to learn a bit more about SCORM and was hoping some nice person out there might be able to point me in the direction of a useful resource (our publish standard at bottom of post). I want to better understand what is sent from SL3 to CSOD and when - I keep reading about terms like .cmi, suspend, normal etc but I'm never quite sure what they actually mean from a practical perspective. I think that a more enhanced fundamental understanding will help me troubleshoot problems better.  

What would be even better would be if I could then understand how the different settings in SL3 affect what is sent and when. 

Any help would be greatly appreciated. We use SL3 & CSOD and publish as SCORM 2004 3rd edition (Complete/Incomplete). 

Steve Flowers
almost 6 years ago05/19/18 at 11:53 am (UTC)

Hey, Chris - 

Here's the complete API reference for SCORM 2004. The second link is a printable poster:

https://scorm.com/scorm-explained/technical-scorm/run-time/run-time-reference/
https://scorm.com/scorm-explained/scorm-resources/reference-poster/

One of the easier ways to see what's going on is to register for a SCORM Cloud account at http://cloud.scorm.com

Using this tool, you can see the log of all of the API calls made in a Storyline module. It's one of the best ways that I have found to troubleshoot a module.  I've pasted an example of a debug log below. Here you see:

  • Initialization of the API. In CSOD, when you click launch, the window changes state to loading. Then a new window is launched with your content. This window uses a unique identifier for the learning object and the user to tell the API where in the database to look for the user's information and where to put it.
  • The first thing Storyline does is get the lesson_mode. This will almost always return as normal. I have seen where some cases returned review but only in older versions of Moodle. This tells the course how to run (browse, normal, or review). I don't think storyline will set anything but normal but I could be wrong.
  • Then Storyline requests the lesson_status. Lesson status is how the LMS knows you're done or not. In a new launch, you'll probably see 'not attempted'. If SL receives not attempted, it will respond to the LMS to set it to 'incomplete'. In CSOD, this is translated to In Progress. 
  • I expanded the line below at 05:32:772 to show what the SCORM Cloud is doing in response to a request.
  • The line you see for LMSGetValue('cmi.core.student_name') isn't something that Storyline requests. I added that in a Javascript trigger. I used that information to display the name of the participant in a text field and also to generate an xAPI agent (another complicated explanation) to report data to an LRS in addition to the LMS and enable a 'trainer's gradebook' of simulation actions that we could not provide in CSOD.
  • This module is set to not retain progress and start fresh with every launch. So you're not seeing any suspend_data calls. After each slide change, you should see an LMSSetValue('cmi.suspend_data') call. This sends the "LMS Cookie" to the LMS so when you resume the SL runtime knows how to suss it out.
  • When the learner has met the conditions for completion (Quiz result, slide count, completion trigger), the score and lesson status are sent from the SL file.  
    + [00:05:47.364] LMSSetValue('cmi.core.score.raw', '100') returned 'true' in 0.001 seconds
    + [00:05:47.365] LMSSetValue('cmi.core.score.max', '100') returned 'true' in 0 seconds
    + [00:05:47.365] LMSSetValue('cmi.core.score.min', '0') returned 'true' in 0 seconds
    + [00:05:47.365] LMSSetValue('cmi.core.lesson_status', 'completed') returned 'true' in 0.001

 

  • And finally, when you exit the module, the session_time is recorded with a call from SL to set that value, the exit state is set to "suspend", LMSCommit is called, as is LMSFinish. This cleans everything up, ensures the right values are sent to the LMS, and closes out the API connection. You can't send anything else after LMSFinish (LMS dependent) until you again initialize the API.

    [23:52:10.198] LMSCommit('') returned 'true' in 0 seconds
    [23:52:10.198] Checking for Commit Error
    [23:52:10.198] Call is error free.
    - [23:52:10.198] LMSFinish('')
    [23:52:10.198] Checking for Finish Error
    [23:52:10.198] Call is error free.
    [23:52:10.198] Close Out Session
    [23:52:10.198] Mode = normal
    [23:52:10.198] Credit = credit
    [23:52:10.198] CompletionStatus = completed
    [23:52:10.198] SuccessStatus = unknown
    [23:52:10.198] MasteryScore = null
    [23:52:10.198] Score = 100
    [23:52:10.198] Sco was taken for credit
    [23:52:10.198] Sco is completed so resetting credit to no-credit and mode to review
    [23:52:10.198] Next entry is normal
    [23:52:10.199] Session Time: PT12M43.35S (76335 hundredths)
    [23:52:10.199] Previous Time: PT0H0M0S (0 hundredths)
    [23:52:10.199] New Total Time: PT12M43.35S (76335 hundredths)
    [23:52:10.199] New Tracked Total Time: PT12M45.98S

Example log from SCORM Cloud

[00:05:29.786] Beginning prerequisites evaluation of activity loan_01
[00:05:29.786] Beginning prerequisites evaluation of activity loan_01
[00:05:29.786] Beginning prerequisites evaluation of activity loan_01_SCO
+ [00:05:32.771] LMSInitialize('') returned 'true' in 0 seconds
+ [00:05:32.771] LMSGetValue('cmi.core.lesson_mode') returned 'normal' in 0 seconds
[00:05:32.771] LMSGetLastError() returned '0' in 0 seconds
+ [00:05:32.771] LMSGetValue('cmi.core.lesson_mode') returned 'normal' in 0.001 seconds
[00:05:32.772] LMSGetLastError() returned '0' in 0 seconds
+ [00:05:32.772] LMSGetValue('cmi.core.lesson_status') returned 'not attempted' in 0 seconds
[00:05:32.772] LMSGetLastError() returned '0' in 0 seconds
- [00:05:32.772] LMSSetValue('cmi.core.lesson_status', 'incomplete') returned 'true' in 0.002 seconds
[00:05:32.773] CheckForSetValueError (cmi.core.lesson_status, incomplete, cmi.core.lesson_status, , )
[00:05:32.773] Element is: lesson_status
[00:05:32.773] Call is error free.
[00:05:32.774] StoreValue (cmi.core.lesson_status, incomplete, cmi.core.lesson_status, , )
[00:05:32.774] Element is: lesson_status
[00:05:32.774] StatusSetInCurrentSession = false, CompletionStatus=unknown
[00:05:32.774] Allowing status change
[00:05:32.774] RunTimeApi_ImmediateRollup called
[00:05:32.774] Transferring RTE data to Activity data
[00:05:32.774] Rolling up activity data
- [00:05:32.774] LMSSetValue('cmi.core.exit', 'suspend') returned 'true' in 0 seconds
[00:05:32.774] CheckForSetValueError (cmi.core.exit, suspend, cmi.core.exit, , )
[00:05:32.774] Element is: exit
[00:05:32.774] Call is error free.
[00:05:32.774] StoreValue (cmi.core.exit, suspend, cmi.core.exit, , )
[00:05:32.774] Element is: exit
- [00:05:32.903] LMSGetValue('cmi.core.student_name') returned 'Flowers, Steve' in 0 seconds
[00:05:32.903] CheckForGetValueError (cmi.core.student_name, cmi.core.student_name, , )
[00:05:32.903] Call is error free.
[00:05:32.903] RetrieveGetValueData (cmi.core.student_name, cmi.core.student_name, , )
[00:05:32.903] Element is: student name
[00:05:32.903] LMSGetLastError() returned '0' in 0 seconds
+ [00:05:32.903] LMSGetValue('cmi.core.student_name') returned 'Flowers, Steve' in 0 seconds
[00:05:32.903] LMSGetLastError() returned '0' in 0 seconds
[00:05:32.925] Beginning prerequisites evaluation of activity loan_01
[00:05:32.925] Beginning prerequisites evaluation of activity loan_01
[00:05:32.925] Beginning prerequisites evaluation of activity loan_01_SCO
+ [00:05:36.152] LMSGetValue('cmi.core.student_name') returned 'Flowers, Steve' in 0 seconds
[00:05:36.152] LMSGetLastError() returned '0' in 0 seconds
+ [00:05:36.152] LMSGetValue('cmi.core.student_name') returned 'Flowers, Steve' in 0 seconds
[00:05:36.152] LMSGetLastError() returned '0' in 0 seconds
+ [00:05:47.364] LMSSetValue('cmi.core.score.raw', '100') returned 'true' in 0.001 seconds
+ [00:05:47.365] LMSSetValue('cmi.core.score.max', '100') returned 'true' in 0 seconds
+ [00:05:47.365] LMSSetValue('cmi.core.score.min', '0') returned 'true' in 0 seconds
+ [00:05:47.365] LMSSetValue('cmi.core.lesson_status', 'completed') returned 'true' in 0.001 seconds
[00:05:47.516] Beginning prerequisites evaluation of activity loan_01
[00:05:47.516] Beginning prerequisites evaluation of activity loan_01
[00:05:47.516] Beginning prerequisites evaluation of activity loan_01_SCO
[00:05:47.668] Beginning prerequisites evaluation of activity loan_01
[00:05:47.668] Beginning prerequisites evaluation of activity loan_01
[00:05:47.668] Beginning prerequisites evaluation of activity loan_01_SCO
+ [00:06:00.778] LMSSetValue('cmi.core.session_time', '0000:00:28.0') returned 'true' in 0.001 seconds
+ [00:06:00.779] LMSSetValue('cmi.core.exit', 'suspend') returned 'true' in 0 seconds
+ [00:06:00.779] LMSCommit('') returned 'true' in 0 seconds
+ [00:06:00.779] LMSFinish('') returned 'true' in 0.001 seconds
+ [00:06:00.780] Pre-evaluation of exit action
+ [00:06:01.231] OverallSequencingProcess for SCORM 1.1 / SCORM 1.2 returned '' in 0.632 seconds
+ [00:06:01.912] OverallSequencingProcess for SCORM 1.1 / SCORM 1.2

 

Steve Flowers
almost 6 years ago05/19/18 at 12:07 pm (UTC)

The example above was for SCORM 1.2. You'll see different data model calls in SCORM 2004. For example, lesson_status is completion_status in 2004.

Part 2 of the long explanation. In a conformant LMS, all of the runtime calls listed in the reference model can be called from the browser using Javascript. This means even if the content (storyline) doesn't have it built in, you could use Javascript to access the API to set or get values. 

Example, if you wanted to add something to the comments data field, you could call LMSSetValue on the cmi.comments_from_learner.n.comment and store away some text. You'd need a way to retrieve that within the module since you cannot report this data in CSOD.

Storyline stores quiz responses in the cmi.interactions data elements. This can be reported using CSOD. There are a lot of cmi-interactions... calls when a question is submitted. Not that only questions associated with the results slide used to track completion of your module will be submitted by Storyline. There are neat things you can do to store values using a quiz question.

Another note, Storyline uses a standard API to submit to the LMS. But the player uses an abstraction of functions to reach that end. If you want to communicate with the LMS directly, you can choose to use the standard LMS API calls in Javascript, or use Storyline's abstracted functions.

Example direct call:
LMSSetValue('cmi.core.score.max', '100')

... and all of the other score setting calls (raw, min)

or Storyline's player functions:

lmsAPI=parent;
lmsAPI.SetScore(90, 100, 0);
SetStatus("completed")

I've found the manual calls to be less reliable and subject to change so much of the time, I have been setting them directly using the standard calls. It's more script but I know it's going to go through even if Articulate changes something in their player functions.

Chris Undery
almost 6 years ago05/22/18 at 2:27 pm (UTC)

So if I've set a course to always restart from where the learner left off (in player) and set the max number of attempts at the quiz to 2 (disable retry quiz button after 2nd attempt) then are you saying every occurence of a question being attempted is being sent to CSOD so I should be able to create a report with trustable question level data? Just want to check your understsanding so I  can run some more detailed tests. We want to start using question level data from elearning but need to be able to trust the data from CSOD.

CSOD told me this:

Setting cmi.exit to “suspend” will ensure that the current attempt is preserved and the run-time data is not reset the next time the SCO is launched. What this means is that when the user finishes the quiz, it suspends their attempt and resets them back to the beginning of the quiz without having them "exit" the course. This means that every attempt of the quiz is being recorded as one attempt as you are asking the course to re-route the user back to the beginning without exiting the course.

Instead, setting cmi.exit="normal" will indicate that the LMS should begin a new attempt with a new set of run-time data on the next launch of the SCO. This should then allow the quiz data to log each attempt.

PS. I have no idea how to change the setting of cmi.exit

Chris Undery
almost 6 years ago06/21/18 at 6:48 pm (UTC)

Hi Chris (great name), I'm still testing this but it is proving far harder than I thought it would be. I've currently got a case open with Articulate as I want to check that they are happy it is sending what it needs to CSOD (to track all quiz and question attempts) so that I can then raise with CSOD if I can't get the reports to show accurate data.

We publish SCORM 2004 3rd edition Incomplete/complete but I think I'm also going to have a go at testing whether SCORM 2004 4th edition or Tin Can has any impact on what CSOD reports.

I've probably spent over 10 working days testing this so far and so far cannot trust the reports that I am pulling out of CSOD. I tracked (spreadsheet) and recorded exactly what questions were answered by whom, when and how often and then looked at various custom reports in CSOD and once you go past one attempt at a quiz it seems to start reporting odd things (& not even consistent.)

It seems unbelievable to me that a simple thing (record all question attempts and answers etc) would be so difficult to achieve and I keep thinking I'm missing something but I don't think I am. 

I want to be able to utilise Storyline for testing in my organisation but I can't if I can't trust the reports. If I want antyhing more than they have passed or failed I can't trust it.

I'll update this thread as and when I get final answers.

Mallory Kudrna
almost 6 years ago07/13/18 at 9:22 pm (UTC)

With the new modern player what browser settings are you using?  I chose Launch player in new window, display window with no browser controls, allow user to resize browser.  As the first option indicates, it creates an odd launch page. I don't love that experience and also received error with reference to API a couples of time.  I am no longer getting that error so I am not able to share exactly what it says.  I do have a modify content in progress unchecking Launch player in new window to see what that experience is like.

Will Findlay
almost 6 years ago07/17/18 at 7:29 pm (UTC)

Hi Julie,

Your question prompted me to test a course that uses the modern player on an iPad pro (hosted in Cornerstone). We don't have many mobile users so I hadn't tested it until now. Unfortunately the results of my testing show that (for me) iPad+Safari+Cornerstone+Storyline Modern Player results in a module that doesn't fit on the screen correctly. You can see here that the module is cut off on the right side:

When I rotate the screen to portrait, I get this (which is even worse):

I suspect it has something to do with how Cornerstone opens the content in a pop-up window, but I'm not sure at this point. So maybe the Modern player isn't ready for mobile when it comes to Cornerstone.

 

Sign In to Reply