Rise courses send inconsistent tracking data to Moodle (or none at all)

Jun 16, 2021

I have a problem similar to Rise progress not saving in Moodle. Some - not all - users progress does not save back to Moodle. Inconsistent results are more prevalent on Chrome on Windows. More than 80% of our users are on desktop-based devices (generally not tablets or mobile) with slow to moderate internet speed. This problem is persistent across a number of Rise packages both big and small. The packages:

  • are published using SCORM 1.2, track using a quiz result, and report 'passed/incomplete'
  • are embedded into the same page (not opening in a new window)
  • have the 'exit course' link enabled, as well as a prominent 'exit course' button on the last page.

This is not a case of data being truncated because of the SCORM suspend data limit. I can measure the suspend data and it's often around 2-3kbytes - significantly less than the SCORM 1.2 limit.

The problems we are seeing relate to the amount of data that is communicated back from the package, which can be everything, some things, or none at all.

Here are some examples of the tracking reports shown in Moodle for attempts.

IE11, Windows 10. The package sends back good data, I can see interactions and there is a good chunk of suspend data.

Valid data from ie11

Chrome 88, Macos Mojave. The package sends back some data. The learner reported to use they were not seeing a completion and has a screen capture of the Rise package on the Quiz screen showing a score of 100% and we matched it to this record using timestamps. There is one interaction recorded - yet the quiz has 8 questions, the suspend data is less than 1k.

 

Chome 91, Windows 10. Another user reporting they didn't get a completion. In Moodle we can see that the attempt is '1' , which we take to mean that Moodle recognises the attempt was started, but in the second screenshot of the Moodle reporting page (which you get to by clicking on the attempt number) there is no data at all recorded and that it thinks the package wasn't even attempted. 

 Just clicking through the reports of randomly selected attempts for many users over many months I find lots of records that have spotty data where some results are recorded such as lesson_location and core.exit but no interaction or suspend data or score. For instance we can use moodle logs to see a user had opened the course and scorm package:

and yet when using the track details report in moodle to reveal what data the package sent back, we see very little in the way of tracking data (Chrome 91, Windows 10).

Yet in the same package, a few records down and a month earlier another user (Edge 44, Windows 10) is a record with all the correct data recorded:

Users often have screen shots of course screens showing their progress or even completion of the packages and yet Moodle has no data to back up those claims. I've looked at:

  • server load / capacity. the server generally has low concurrency. the issues do not seem to be related to time-of-day.
  • testing the packages externally, generally on scorm cloud but also independently in other players or other Moodle installations.
  • asking the learners to do internet speed tests. some report getting a 'very slow' to 'good' result from these, consistent with most experiences with  Australian internet.
  • asking users about any potential browser extensions or security apps that might be interfering. Learners generally either don't know how to check or are unable to change anything about their environment (e.g. office & library computers are often locked down).

Things I would like to know about how SCORM works in Rise packages when used within Moodle:

  1. Does a Rise course just shout out its information and hope a server is listening? That is, does it listen to ask 'Did you get that' or just hope?
  2. Does it record back as the learners navigates through pages or does it only send all the data when the user presses the Exit Course link?
  3. Does Rise handle the internet dropping out while the learner is navigating only to have it reconnect before their finish the course, and what happens to the data in that intervening period? Does it retry sending it?
  4. Are other people seeing similar inconsistencies with their Moodle installations and what versions and server environments are they using?
  5. Is there way to make the Exit Course button more prominent? Many learners have indicated they didn't know it was there because it is small text and grey and sometimes isn't able to be seen depending on the page backgrounds. I have included below the Exit course button I have on the last page which has a higher click rate - but still some users simply close their browser tab when they are done without exiting the course using either link.

I'm hoping that when a user presses a link or performs an interaction, Rise immediately notifies Moodle of the change rather than waiting until the end, as it would allow for dropouts (common), server lag (highly uncommon), or user error (common).

3 Replies
Tim St. Clair

Cheers, I got a detailed reply from Support. The short of it is to keep Moodle up to date as a patch to alleviate some of the issues related to closing the package and how it saves data during that close - which is a problem affecting Chrome and Edge but not other browsers at this time. Moodle devs discussed and patched this some time ago (in a version higher than my client runs of course - time to convince them to upgrade) - here is the patch https://tracker.moodle.org/browse/MDL-67175 and discussion https://moodle.org/mod/forum/discuss.php?d=404312 for reference. 

It's confirmed that Rise is only signalling back the bulk of the completion data only upon course exit, rather than "as you go" and that there isn't any graceful handling if the network drops out and comes back later (which is surprisingly common, even on physical networks); this may change in the future.