Blog Post

Product Updates
7 MIN READ

Engineering Journal: Storyline Quality Update Q4 2024

JasonFamularo's avatar
28 days ago

Hello, E-Learning Heroes community members! I’m Jason Famularo, a Senior Software Engineer on the Storyline team. I’m here to give the quality metrics update for the fourth quarter of 2024.

Quality Metrics

Storyline captures a variety of performance and quality metrics to assist the engineering team with finding, troubleshooting, and resolving crashes and performance issues. This data collection is optional and can be toggled on or off. When toggled on, the app sends telemetry and crash data that helps us make the product better (and provides the details for the graphs below). When you encounter an error dialog, providing more context and details helps us to narrow down issues.

This toggle can be found in the Preferences of the Articulate 360 app on your PC.

You can always open a support case with our wonderful support team. Attaching a .STORY file that demonstrates the issue is one of the best ways to assist us in helping you.

One quick note before we jump in: we recently adjusted some of the thresholds for our quality metrics. After reviewing the data for the past 18 months, we decided to change a few of our goals to be more consistent with long-term trends that would allow us to better balance quality work with everything else that goes into making Storyline great.

Let’s review our quarterly quality metrics!

Application Error Rate*

The application error rate measures how often Storyline displays the “Articulate Storyline Error Report” dialog. We track this data for Storyline sessions—or unique instances of opening and then later closing Storyline. Our goal is to get this metric under 3%.

The application error rate for Storyline sessions was hovering around 1.15% and jumped up in Update 91. We’ve been working successfully on bringing that metric back down.

Storyline’s application error rate per session for updates 87 through 94.
Application Error Rate - Storyline Sessions

Line chart depicting the Storyline application session error rate from update 87 through update 94. The Y axis is the error rate percentage and the X axis is Storyline updates.

The data points indicate:

  • Update 87 released in April 2024 was 1.12%.
  • Update 88 released in May 2024 was 1.16%.
  • Update 89 released in June 2024 was 1.16%.
  • Update 90 released in July 2024 was 1.11%.
  • Update 91 released in August 2024 was 1.89%.
  • Update 92 released in September 2024 was 1.78%.
  • Update 93 released in October 2024 was 1.67%.
  • Update 94 released in November 2024 was 1.42%.

*Previously, we reported on the error rate for users per release, but that metric didn’t fit with the other metrics we report on. We still track it internally, as it is a good leading indicator of new problems; we just won’t report it in our Engineering Journals.

Downgrades

This metric tracks how often a Storyline user updates to a new version of the application—only to downgrade later to an earlier version. We interpret downgrades as an indication that authors encountered issues in a new version that prevented them from completing their work.

Last year, we saw this metric dip below 1% at the end of the second quarter and remain there through the middle of the third quarter. It has climbed as we worked to remove old third-party dependencies in our Modern Player. We’ve addressed these issues with a series of service releases and, as expected, downgrades returned to a sub-1% level. Collecting feedback when users downgrade to a previous version of Storyline was very helpful in pinpointing the Modern Player issues to work on.

Storyline’s downgrade percentage for updates 87 through 94.
Application Error Rate - Storyline Sessions

Line chart depicting the Storyline application session error rate from update 87 through update 94. The Y axis is the error rate percentage and the X axis is Storyline updates.

The data points indicate:

  • Update 87 released in April 2024 was 1.12%.
  • Update 88 released in May 2024 was 1.16%.
  • Update 89 released in June 2024 was 1.16%.
  • Update 90 released in July 2024 was 1.11%.
  • Update 91 released in August 2024 was 1.89%.
  • Update 92 released in September 2024 was 1.78%.
  • Update 93 released in October 2024 was 1.67%.
  • Update 94 released in November 2024 was 1.42%.

Defect Rate

This metric tracks the percentage of open support cases associated with an unfixed bug. An increase in this number signals that our support team is spending time fielding bug reports instead of helping customers get the most out of our products, so our goal is to keep this value below 10%

This metric has been below the 10% threshold for quite some time now, but got pretty close as we worked on issues relating to the Modern Player dependency removal discussed in the previous section.

We rely on support cases to direct our bug-fixing efforts, so I encourage you to contact our support team if you’re experiencing issues with Storyline.

Storyline’s defect rate for April 2024 through December 2024.
Defect Rate

Line chart depicting the Storyline defect rate from April 2024 through December 2024. The Y axis is the defect rate percentage and the X axis is the month.

The data points indicate:

  • April 2024 had a defect rate of 5.48%.
  • May 2024 had a defect rate of 6.55%.
  • June 2024 had a defect rate of 9.48%.
  • July 2024 had a defect rate of 8.78%.
  • August 2024 had a defect rate of 10%.
  • September 2024 had a defect rate of 7.12%.
  • October 2024 had a defect rate of 7.45%.
  • November 2024 had a defect rate of 6.75%.
  • December 2024 had a defect rate of 7.2%.

Publishing Failures

This metric tracks the number of users who get an error during publishing. We’ve made an effort this year to address the top publishing failures. While we haven’t seen a significant drop in these error percentages yet, we expect to see one soon because adoption of a new version of Storyline often takes a few months.

At the end of last quarter, we changed two aspects about how we track publishing errors. The first one I explained in the last journal: we were unfairly penalizing ourselves for publishing attempts that were canceled. We changed a second aspect of this metric: consider publishing attempts that send “start” telemetry but don’t send “finished” telemetry as a failure. We would consider that a sign that the application crashed during publishing. Unfortunately, there were two small bugs at the end of the publishing process that caused us to not send the “finished” telemetry even though the publishing finished. The previous quarter’s journal included the second aspect, but this chart does not. We estimate those would have added 0.5% to these numbers (to make the average closer to 2% than to 1.5%).

Our goal is less than 3%, so we are doing good here!

Storyline’s publishing failure percentage for updates 87 through 94.
Defect Rate

Line chart depicting the Storyline defect rate from April 2024 through December 2024. The Y axis is the defect rate percentage and the X axis is the month.

The data points indicate:

  • April 2024 had a defect rate of 5.48%.
  • May 2024 had a defect rate of 6.55%.
  • June 2024 had a defect rate of 9.48%.
  • July 2024 had a defect rate of 8.78%.
  • August 2024 had a defect rate of 10%.
  • September 2024 had a defect rate of 7.12%.
  • October 2024 had a defect rate of 7.45%.
  • November 2024 had a defect rate of 6.75%.
  • December 2024 had a defect rate of 7.2%.

Incomplete Sessions

This metric tracks how often Storyline quits unexpectedly due to an error. Our goal is to maintain this metric under 3%.

The Storyline team spent Q1 focused on improving this metric as much as possible. The percentage peaked at 3.8% at the end of Q4 2023, and those efforts have slowly made a big difference, as the percentage has consistently, yet slowly, dropped in 2024. We’ve finally hit our goal and will do the work needed to keep this metric on target.

Storyline’s incomplete session percentage for updates 87 through 94.
Incomplete Sessions

Line chart depicting the Storyline incomplete sessions from update 87 through update 94. The Y axis is the incomplete session percentage and the X axis is Storyline updates.

The data points indicate:

  • Update 87 released in April 2024 was 3.19%.
  • Update 88 released in May 2024 was 3.06%.
  • Update 89 released in June 2024 was 3.06%.
  • Update 90 released in July 2024 was 2.98%.
  • Update 91 released in August 2024 was 2.93%.
  • Update 92 released in September 2024 was 2.91%.
  • Update 93 released in October 2024 was 2.99%.
  • Update 94 released in November 2024 was 2.88%.

Wrap-Up

Storyline, like any big project, is akin to running a marathon. You make slow and steady progress at achieving your goal, but sometimes you stumble and fall down. Stumbling throws you off, and you have to refocus and recommit to running the race, knowing that your pace for that particular mile will suffer. Your goal is still to finish the race, but it will take a little bit longer to achieve that goal. We’ll meet our goal with the quality metrics—a couple of stumbles or roadblocks won’t keep us from finishing.

In the meantime, if there are any topics you’d like to see covered in these quality updates, please reach out to the team at storyline-engineering@articulate.com.

Updated 28 days ago
Version 1.0
No CommentsBe the first to comment