Engineering Journal: Storyline Quality Update Q1 2024

Hi E-Learning Heroes community members! 

I’m Jesse Taber, engineering manager for Storyline 360, here to give you an update on the quality work Articulate CTO Kerry Munz kicked off in April 2023 after hearing from users that Storyline 360 quality wasn’t up to par. The initial plan (outlined in this discussion: Storyline Under a Microscope: An Update From the CTO) was to have the engineering team pause work on other projects and focus solely on quality and stability for six months. During that time, I shared monthly updates on our progress in that discussion thread.

In my final monthly update, I promised we would continue to focus on quality and stability in Storyline 360, even as we restarted work on enhancing and extending the product. I also committed to providing regular updates in a quarterly engineering journal. So here I am, making good on that promise. 

Each quarter, I’ll share an update on the quality metrics we’ve been working to improve along with any insights we’ve learned along the way. 

Application Error Rate

The application error rate measures how often Storyline 360 displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions—each session is a unique instance of opening and then later closing Storyline 360—and users. 

In the third quarter of 2023, approximately 1.5% of Storyline 360 sessions encountered an error. That number dipped below 1.5% and started approaching our goal of less than 1% in the fourth quarter.

Line chart showing Storyline 360’s application error rate per session for updates 78 through 84.

We track the percentage of Storyline 360 users who encounter one or more errors in each release. Back in September 2023, I reported that about 15% of Storyline 360 users had encountered an error when using Update 79. Unfortunately, that number ticked up to 20% as more users upgraded. Since then our efforts have this metric trending in the right direction, as the chart below shows.

Line chart showing Storyline 360’s application error rate per user for updates 78 through 84.

Progress on the application error rate has slowed as the team has devoted resources to other areas, but we’re pleased with improvements so far and will continue working in this area.

Downgrades

This metric tracks how often a Storyline 360 user updates to a new version of the application—only to downgrade later to an earlier version. We interpret downgrades as an indication that authors encountered issues in a new version that prevented them from completing their work.

Last year, we saw this metric dip below 1% at the end of the second quarter and remain there through the middle of the third quarter. Since then, it has climbed and seesawed between 1% and 1.6%, with Update 84 dipping under 1%. While we’re happy to have reduced this number from the 2% we saw last year, we want to understand better why our customers downgrade. We’ll be working to clarify this in the coming months.

Line chart showing Storyline 360’s downgrade percentage for updates 78 through 84.

Defect Rate

This metric tracks the percentage of open support cases associated with an unfixed bug. An increase in this number is a signal that our support team is spending time fielding bug reports instead of helping customers get the most out of our products, so our goal is to keep this value below 10%. We achieved that goal in the latter half of 2023 and continued into 2024. We’re also committed to addressing sudden spikes in this metric, even if they don’t exceed our 10% threshold. 

For example, this value reached approximately 8% in August, then again in December and January. If a Storyline 360 release introduces a new bug, we often see a spike in this metric as customers open support cases. Any bug introduced in the past four releases is designated a “recent regression.” We are constantly evaluating our test processes to prevent regressions like this, but when they occur we prioritize fixing them quickly to minimize the impact they can have on our customers.

We rely on support cases to direct our bug-fixing efforts, so I encourage you to contact our support team if you’re experiencing issues with Storyline 360. 

In addition, customers who participate in our private beta program can report issues they find directly to the Storyline engineering team and see new features before general public release. If you’re interested in participating, contact us at beta@articulate.com. 

Line chart showing Storyline 360’s defect rate for July 2023 through February 2024.

Publishing Failures

I shared this metric for the first time in my final monthly update last September. We initially saw that nearly 6% of publishing operations were failing. We reduced that number to 2% in June (not shown in the graph below), but it later climbed to about 5.5%. Since then, it’s settled just north of 4%. We recently updated how we track publishing failures to provide more insight and help reach our goal of less than 1%.

Line chart showing Storyline 360’s publishing failure percentage for updates 73 through 79.

Incomplete Sessions

This metric tracks how often Storyline 360 quits unexpectedly due to an error. The last time I shared this metric, it hovered around 6%, but we suspected that Storyline 360 might sometimes fail to correctly record an event for the end of a session, creating false positives in the data that were inflating the number. In Update 80, we rolled out a change aimed at improving the accuracy of this data and found that closer to 4% of sessions experience this issue. Our goal is to drive this metric below 1%, because having Storyline quit unexpectedly is extremely frustrating for you and unacceptable for us.

Fixing issues that lead to incomplete sessions can be difficult because the data we collect for unexpected errors sometimes can’t be captured before Storyline 360 shuts down. We are considering ways to improve our understanding of how and why this happens. In the meantime, opening a support case for instances of Storyline 360 quitting unexpectedly could help us get to the root cause of these issues.

Line chart showing Storyline 360’s incomplete session percentage for updates 78 through 84.

Wrap-Up

I hope you’ve found this inside look at how the engineering team is working to improve quality in Storyline 360 insightful. If you enjoyed it, be sure to check out my deep dive into 64-Bit Storyline for more behind-the-scenes information from engineering. 

Stay tuned for next quarter’s quality update! And please email me at jtaber@articulate.com if there are specific topics you’d like me to cover.