Forum Discussion
Storyline and Workday Learning issue with reporting correct score
We have a very frustrating scenario with storyline and workday, so im hoping someone may have also experienced the same issue and can help!! We have created a 60 question quiz in storyline with a pass rate of 60% , there is a timer of 30 mins. The SCORM is published in 2004 4th edition.
Workday records the score correctly when the users complete the quiz, but the next day when checking the scores they are different in workday than what they are in the SCORM file. For instance an employee scores 75% which workday picks up, but then a few hours later it will say a different score such as 68%. Workday are adamant it's the SCORM file but when reaching out to SCORM cloud for advice they have said in order for the course to change status in an LMS, the course would need to be actively in session. I have uploaded the courses to SCORM cloud and so far they haven't changed scores which makes me think its the LMS
However when we have used Adobe Captivate for a similar quiz, we do not get this issue, so I can see why workday are saying it's the SCORM, but I don't know what we need to change in order to make it work. The settings are the exactly the same as the storyline file in terms of completion the only difference is that storyline gives us a "pass/fail" "complete/incomplete" etc option.
Has anyone had anything similar to this and if so what was the solution?
17 Replies
This is 100% correct "SCORM cloud for advice they have said in order for the course to change status in an LMS, the course would need to be actively in session". It was the first thing I thought when reading your issue. There is no way the course can impact the score if it has not been launched again as it needs to be actively running a session in the browser.
I'm wondering if the LMS is running some kind of overnight job that is affecting the score.
The only thing I could think that could be different between a course that doesn't have the problem and one that does would be a mastery_score being set on one course and not on the other. I'm really reaching here, as it's definitely an LMS thing. The only reason I mention mastery_score is because this can impact whether an LMS will take control of completion, but it should definitely not take control of the score.
I think the onus should be back on the LMS provider here to track an instance of this occurring. Do they have event logs that can see a score changing from X to Y. What time it changed, what made the change?
I always recommend SCORM Cloud as the benchmark for testing content, and am always confident when using it to determine if an issue exists on the content or LMS side.
On last tip, I would always take a collaborative approach with LMS admins. You will get to a solution quicker if you work together.
Hi Sam,
Thank you very much for this information and advice! We have been collaborating with the lms admin and workday but just nothing seems to be going forward apart from “it’s the scorm file” it feels like we have tested the file in every single scenario possible!
I have checked scorm cloud and the score hasn’t changed on there so I would also say it could be the workday lms not communicating with the articulate course correctly.
However it does seem strange that when a course is published on adobe captivate we do not have this problem? The mastery_score is not something I’m familiar with but if this is something we can change in the settings of the scorm? Although like you said I’m not sure why the lms would still read a different score to what the users have actually achieved.
I will go back to workday with this information and see what they respond with!
Good luck Natalie. I'll keep an eye out for an update on here. I think the key thing to keep pushing with Workday is that the score is updating after the course has been closed and before it has been launched again. During this period of time, the course cannot impact any data on the LMS.
Hello,
We also encountered this problem with a Storyline course published in Scorm 2004 4th edition.
The course results changed the next day.
For the moment, our solution is to publish them with Rise since the problem is only with Storyline.
We'll be working with Workday to find the cause, but if anyone can help, we'd appreciate it.
Thank youHi Claude!
Here is some advice we got from SCORM cloud
"Your course does a "running average" during the assessment...for example, if you get the first one right, the course sets 100. If you miss the 2nd, the course sets 50. When I get the 3rd one right, I am at 2/3, the course sets 66. There is nothing wrong with what your course is doing. It is just calculating the score in a way it wants to. On the other hand, there is not technically anything wrong with what Workday is doing. They are modifying the reports in some manner post runtime with their ETC functionality, but they are just looking to make sure they get the right score which they believe is the highest score."
Since the SCORM file seems to send an average score for each interaction submitted when in workday, we set the course to submit all of the interactions simultaneously at the end of the quiz Here's how to set that. and this seems to work!
When using navigation buttons instead of a submit button on your quiz, it's possible that learners can skip questions and submit the quiz without realizing they missed questions, so you may want to warn them before submitting the quiz, as described here.Technically workday and articulate are working as they should, but this is the work around to ensure workday and articulate work with each other.
A bit frustrating as it doesn't seem to happen in other LMS's. Hope this helps!
I am a bit frustrated myself as I have run many test on Workday to fix the same issue. This is now my 3rd project where we are running into this issue using standalone and question banks. So far the only successful project is the one not using any feedback slides.
I've been following the scores of learners for this reason. Some scores hold for a couple of days then reset to 100% or others would change to 100% in a couple of hours of completion. I do know the learners are not going in again to retake the lessons as I would see a double enrollment.
I would like to try what is suggested above of submitting the interactions all at the end. I don't understand how to as I designed my quiz using the free form question using feedback and need a submit interaction trigger on each slide in order for the review button on the result slide to identify the correct and incorrect response.
Any suggestions would be helpful. We also publish using 2004 4th edition pass/failHello VoulaBalkos,
Thanks so much for reaching out to share that you’re also experiencing issues with your Storyline course in Workday. This is the data Storyline sends when publishing for SCORM 2004, so it is concerning when an LMS isn't reporting what is being shared.
Based on what others have shared, it sounds like this issue stems from how Workday is reporting the data it receives. When we see people running into an issue with their LMS, we recommend testing the course in SCORM Cloud to solidify if the issue is within the course or the LMS. I would recommend testing your course in SCORM Cloud so we know what path to take!
If you're still seeing this issue in SCORM Cloud, I'd be happy to assist you with modifying the submission method for your standalone question slides. This can be done, but I want to confirm that modifying this will remove the feedback.
- If your question slides aren't in a question bank, switch to Story View.
- Select all your question slides. Use Shift+Click for consecutive slides or Ctrl+Click for non-consecutive slides.
- Use the Slide Properties panel in the lower right corner to mark the Prev/Next boxes and uncheck the Submit box. All selected slides will be changed at the same time.
Did this do the trick? If not, the next step is for us to examine your project file! Please upload your .story file to this thread or privately in a support case when you have the chance.
Looking forward to hearing from you!
Thank you Luciana for your suggestions. I will look further into the build. I have the tendency to use the built-in free form pick one slide templates and usually use a shape as my submit, Pre/Next button. In this case the submit button has been deselected from the player features as well as each slide properties along with the prev/next.
I will further investigate and look into your suggestions.
Many thanks!
Hi! I am running into this issue with Workday and Storyline. Did you ever find a fix?
Hello KellyKreps-0451,
I understand that you are also experiencing reporting issues with your Storyline 360 course in Workday. Happy to learn more!
- Is the publishing setting set to SCORM 2004, or are you using another publishing setting?
- Are you experiencing the same behavior when testing the course in SCORM Cloud?
- Is this happening with any other courses you've published to Workday, or just this one?
We'd be happy to take a closer look at your .story file so that we can continue troubleshooting. You can share your file in a support case.
Looking forward to hearing from you!
Hi Luciana, the file is SCORM 2004, no it does not happen in Cloud, and it's happening with all of our Storyline assessments. We are working in Workday support on this as we have narrowed down the issue to how their system runs overnight.
Hello KellyKreps-0451, I submitted support case #01332676 a few weeks ago regarding an issue with Workday and Storyline SCORM/2004, specifically related to the question bank and the functionality for retrying only missed questions. Recently, I encountered a new issue with the same SCORM file—the score unexpectedly changed from failing to passing, and in some cases, from passing to 100% within a few days. It appears that other community users are facing similar challenges. Has a solution been identified for these issues?
No, there has not been a fix that we have found. We think it has to do with multiple results slide within our assessments that lead to one combined results slide- is that how yours is set up as well?
Thank you KellyKreps-0451 for your quick response. Could you clarify what you mean by having multiple results slides within the assessment? I currently have only one results slide, the user has just one retry attempt, and the trigger is set to show only the missed questions.
We have built the assessment to be in Spanish and in English, so both languages go to their own results slides and then continue to a combined results slide. Just wondering if we had similar issues, but it sounds like maybe we don't. What we have figured out is that when the assessment is being graded, the score doesn't just add the points for a correct answer, but averages the score based on numbers of questions answered correctly. (Instead of 10 + 10 - 10 + 10 + 10 = 40/50, it says 1/1 correct 100%, 2/2 correct, 100%, 2/3 correct 66%, etc.) Because of the way Workday runs overnight, it for some reason pulls the highest score reported from the quiz from any time within the quiz, not from the final answer. We have not found out why this is or how to fix it on our end. Hope that makes sense!
Hi KellyKreps-0451 . The only way I have found that you can stop incremental scoring is by not triggering the "submit interaction" trigger on each question. I delete the trigger and replace with either a "jump to slide next slide" or add logic to show the appropriate correct/incorrect feedback.
The questions are processed without any issues on the results slide, and this is when the score is written to the LMS.
This is a simple solution if you don't have feedback layers, as the submit trigger can just be changed to "jump to slide next slide".
If you do have feedback layers on the questions, this is more laborious, as the logic to show the correct/incorrect layer would need to be added to each submit button e.g:
I think the Workday score compatibility issue has been around for a long time and are not likely to change it.
When I used to build custom eLearning, we wouldn't write a score in this way, we would always write the score to the LMS when all questions had been completed and we showed the user the results. I think it would be a good configuration option for Storyline assessments to be able to choose when the score is written to the LMS (after question is submitted, or at the results slide).
