Storyline 2: Assessment Retry Completion issue

Sep 17, 2014

Ian Jaffe
By Ian Jaffe

Hi there,

I am doing some basic testing of Storyline 2 for our company to get a feel for what has changed in the new version. In doing so I have encountered what seems to be a rather significant bug in the SCORM 1.2 completion tracking when an assessment has the retry button on the results slide. Basically what is happening is that if the user fails the assessment on a first attempt as soon as they hit the retry button they are marked complete.

The course I am testing is a very simple default course created directly in Storyline 2. It has two content slides and a question bank with five true false questions:

The results slide is set to an 80% passing score and to allow the user to retry the quiz:

The reporting and tracking options in the publish settings are set to report an LMS Status of Completed/Incomplete and track by quiz results:

Our LMS is a SumTotal LMS that prompts the user with a popup as soon as they are marked complete. I take the course and get a failing score of 20%. At this point on the Resulst Slide I see the correct score and I hit the

retry button:

As soon as I hit the retry button I am taken back to the first question of the assessment and then quickly prompted that I have been marked complete by the LMS:

In the LMS when I exit the course I can see on the progress summary screen that the lower line item for this attempt is set to a status of complete and score of 0%, but the General overall score is set to 100%:

The activity details shows that same 100% overall score, but shows that I answered 4 of the 5 questions incorrectly which should be a 20% score as was shown on the results slide:

If I exit the coourse from the Resulst Slide immediately I am correctly marked as having a 20% score and incomplete status. So the error definitely appears to be tied to the retry functionality.

And since I know from experience support is going to ask this, yes we did test this in SCORM Cloud as well and saw the type of error there. Using the same course package and following the same steps where I fail, hit retry, and exit the course on the first question of the retry I see the following results in SCORM Cloud where it indicates I am complete with a score of 0%. This should be incomplete with the 20% score I got on the first attempt.

The detailed activity report in SCORM Cloud has some additional information that shows that after the first assessment attempt is taken the course updates the raw score 7 more times. Incrementing it from 20, to 25, to 33, to 50, and finally to 100% where it marks me complete. What is causing these additional adjustments to the raw score?

I have also created a quick and dirty screencast showing the behavior on both environments:
http://www.articulate.com/support/contact/screencast.php?screencastid=2dd8e70a0bf84a3f92ce2ebbd08264bf

And the .story file is attached as well.

Honestly how does something this basic being broken make it into a final release? This is a showstopper in terms of us using Storyline 2.

26 Replies
Justin Grenier
over 9 years ago09/17/14 at 1:42 pm (UTC)

Thanks for the comprehensive explanation of the problem your are experiencing, Ian.  ...and thanks in particular for (1) testing for reproducibility of the problem in SCORM Cloud and (2) taking a look at SCORM Cloud's debug logs in an attempt to observe what is happening under the hood.

I have created Articulate Support Case Number 00422236 on your behalf  and escalated the Case directly to our Tier 2 team so that we can spend some time testing for reproducibility of the problem.  Expect to hear from us soon.

Thanks again!

Justin Grenier
over 9 years ago11/21/14 at 10:23 am (UTC)

Good Morning, Yvonne.

We were able to duplicate the issue Ian reported.  We did find that the issue seemed to be specific to Question Banks, and that as such, directly adding Question Slides was a possible temporary workaround.

We have reported this issue to our Quality Assurance team for their review.  I cannot offer a time frame for when this issue will be addressed, but we will keep this Forum Thread updated with any new developments.

Please let us know if you need anything else, and have a great day!

Heinz Benninghoff
over 9 years ago11/28/14 at 8:50 am (UTC)

Hi Ian

The problem that you are experiencing is not a Storyline bug that makes your results on your LMS go to 0%, the retry button itself has a trigger on it that resets your results when you click on the button. This means that when you click on the retry button and close your assesment it will definitely show a 0% pass score, if you retry the assessment it automatically makes your results to a 0% pass score until you finish the assessment to get the score fo lets just say 50% then it will give you a pass score of 50%.

It is the retry buttons soul purpose to reset the results page. I have experienced this through all the courses I have developed.

Heinz Benninghoff
over 9 years ago11/28/14 at 8:53 am (UTC)

Hi Ian

The problem that you are experiencing is not a Storyline bug that makes your results on your LMS go to 0%, the retry button itself has a trigger on it that resets your results when you click on the button. This means that when you click on the retry button and close your assesment it will definitely show a 0% pass score, if you retry the assessment it automatically makes your results to a 0% pass score until you finish the assessment to get the score fo lets just say 50% then it will give you a pass score of 50%.

It is the retry buttons soul purpose to reset the results page. I have experienced this through all the courses I have developed.

Justin Grenier
over 9 years ago01/06/15 at 11:27 am (UTC)

Good Morning, Ian.

I wanted to briefly revisit this thread to note that we've documented this as a known issue and that it'll be fixed in a forthcoming software update for Articulate Storyline.

In the meantime, you can avoid this behavior by copying and pasting the questions directly into your course rather than using question banks.

Mark Whitefield
over 8 years ago12/13/15 at 9:13 pm (UTC)

Hello there - Has this bug been fixed? - the issue that I am having is that when users get a failing score and then click Retry Quiz - if they exit out of the module without completing it - the LMS over-writes their initial score reports 100% complete and 100% score - I see that a fix is to go back and move all of the questions out of the Question Banks into the main body of the module - but was hoping that I would not have to do that - Thanks

Natasha Bomba
7 years ago02/03/17 at 7:47 pm (UTC)

Any updates on this? I'm also having the same issue. After racking our brains trying to figure out why we were getting 100% completion with errors in our assessment, it looks like this is a known issue. I've got two assessments, with both results going to a final slide. We've tried every configuration--add the results, pass each test, have the final slide score each slide (instead of the results slides) and we keep getting the 100% on the retry versions. We are also on SumTotal. And I can't send files because of confidentiality issues and firewalls.... Also, I am not using a question bank and no, I cannot test to Scorm cloud either for the same reason I can't send files...

Leslie McKerchie
7 years ago02/03/17 at 7:57 pm (UTC)

Hi Natasha - Is this the issue we discussed here? I wasn't aware that a Re-try was a part of your equation?

As you can see from Justin's response above, the issue specified here was corrected in an update. I was also able to see where Mark submitted a support case and our team was not able to re-create the issue, a published file was shared, and we did not hear back from the user. So, I have no update outside of what you are seeing here.

Leslie McKerchie
7 years ago02/03/17 at 8:33 pm (UTC)

Hey Natasha - I went ahead and tested the same scenario we went over in the other thread and added in the retry and I'm still not experiencing the behavior of the score reporting as 100% after retry either.

 

Published output:

https://cloud.scorm.com/sc/InvitationConfirmEmail?publicInvitationId=819c1fd3-7d8b-4791-a16c-88d2e883d6af

File attached.

Natasha Bomba
7 years ago02/03/17 at 10:37 pm (UTC)

Hi Leslie,

 

Yes it was. I didn't mention the retry because I didn't realize that was part of the issue. We may have resolved it though. It has to do with the reporting. So what I've done is put the Submit Results trigger on the success layer instead of the main slide. That means that results calculation won't be complicated by a learner's previous attempt. We'll assume that people won't do the quiz twice if they pass! It all seems strange to me as I would thing that there shouldn't be anything sent to the LMS if learners are retrying the quiz.

Jacques Theron
7 years ago03/01/17 at 11:32 pm (UTC)

Good day,

I have a similar issue, but my problem is not getting the course completed, it is the 'Retry Quiz' function that does not allow the user to retry the quiz. I have selected it in the Results page properties, but when I click the 'Retry Quiz' button (whether I have passed or failed), the 'Your Score' resets to 0% and nothing else happens. The quiz is locked and even on the review it has cleared all my responses. 

I have unselected the Retry Quiz function, republished, then selected the retry function again and published again, but still, this issue remains. 

Reporting is set to Passed/Incomplete

I have published the module for LMS and is testing it in a SCORM1.2 player, it will be published on Cornerstone LMS.

Any insights?

Thank you and kind regards,

Jacques 

Wendy Farmer
7 years ago03/01/17 at 11:38 pm (UTC)
Jacques Theron

the 'Your Score' resets to 0% and nothing else happens.

Do you mean it doesn't jump back to the quiz slides to let you try again?

The triggers on the Retry Quiz should be in this order:

  • Reset results
  • Jump to xx (first quiz) slide

Also check that your quiz slide properties are set to 'automatically decide' and NOT 'reset to initial' state.

This discussion is closed. You can start a new discussion or contact Articulate Support.