Forum Discussion
Storyline under a Microscope: An update from the CTO
Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where thing...
A Year of Change: Reflecting on 12 months of Storyline development
Since stepping into the CTO role at Articulate in early 2023, I've had the privilege of diving deep into our customer experiences. Back then, it quickly became clear that while there was a lot of love for Storyline, there were valid frustrations around unresolved bugs, lack of transparency, missing 64-bit support, and insufficient communication from our engineering team.
Hearing you loud and clear
About a year ago, your feedback was unmistakable: we needed to listen better. I shared our engineering teams’ commitment to change in a post titled, “Storyline under a Microscope: An update from the CTO”, and I'm thrilled to talk about the strides we've made since.
Our Shift in Focus
Today, our conversations with customers are future-focused, revolving around quality, advanced capabilities, and accessibility. It’s a shift from frustration to anticipation, and while we haven’t solved every issue, the progress is undeniable. So let’s review the key themes from the past year and highlight the progress we made on the journey.
Reflecting on Key Themes: Quality, Transparency, and Communication
Quality became focus number one for our team. In a nutshell, we hit pause on all new features to zero in on fixing existing issues, aligning our teams more closely for better quality control, diving into real customer scenarios to guide our releases, and doubling down on modernizing Storyline, including launching a 64-bit version in beta. In more detail:
- Prioritizing Quality Over Features: We took a strategic pause on developing new features for Storyline for six months, dedicating this period solely to enhancing product quality. The entire Storyline Engineering Team shifted their focus to addressing and resolving existing quality issues, ensuring a stronger foundation for our product.
- Unifying our Approach for Better Results: Previously, our Engineering, Quality Assurance, and Support teams worked independently on Storyline releases, which sometimes led to siloed efforts. We've since revamped our approach, bringing these teams into a cohesive unit. This collaboration ensures we can identify and address quality issues more efficiently, resulting in faster and more effective solutions.
- Learning Directly From User Experiences: In partnership with David Anderson, our Director of Customer Training, we've integrated real-world course-building scenarios into our testing process. These scenarios, reflective of our customers' daily challenges, are now a crucial part of validating each Storyline update before it goes live. This practice has not only improved our issue detection but has also deepened our understanding of what our users truly need.
- Modernizing Storyline: A year ago, we reinforced our commitment to Storyline by launching the first 64-bit version, Storyline 360 x64, in a public beta. We're now in the final stages of preparing Storyline 360 x64 for widespread release and are actively working on additional modernization projects to ensure Storyline continues to meet the evolving needs of our users.
Along with our investment in quality, we recognized the need to increase transparency to clarify how we make decisions around what we work on.
Commitment to Transparency
Starting last year we committed to being more transparent about how we approach building and maintaining Storyline. We opened up about our decision-making process, our progress toward reducing unexpected errors, and our plans for addressing long-standing bugs. Our revamped bug triage process ensures we're responsive and focused on what matters most.
- Reducing the error rate: When we started this effort in April of 2023, about 3% of Storyline sessions were encountering an unexpected error. We committed to getting this metric under 1%. As of March 1, 2024 1.2% of Storyline sessions are encountering an unexpected error. We’ve worked hard to reduce these errors and we won’t stop until we get under our 1% goal because any error that interrupts your day is like a “paper cut” in your way.
- Staying “below five”: Twelve months ago, we discovered we had stopped paying attention to “older bugs”. So we focused our attention on not just recent issues but all bugs that had five customers or more. We have 100s of thousands of users, and set a line in the sand that all bugs must stay “below five customers” otherwise we stop feature development to fix. We cleaned up our backlog of bugs affecting five or more customers and Storyline is much stronger for it.
- Revamping our bug triage process: Our bug database was, admittedly, a bit of a mess in April of 2023 and it was difficult to find signal in the noise. The right things were not always getting prioritized. So along with jumping on those older bugs, we revamped our daily bug triage approach to prioritize new bugs in the latest releases. We also started tracking the overall defect rate coming in from support cases to drive it below 10%. These changes helped us jump on issues causing pain quickly so that it didn’t spread.
To reinforce our quality and transparency commitment, we recognized the need to increase communications to ensure your concerns were being heard and addressed.
Emphasis on continuous communication
In summary, monthly updates last year have kept you informed. We’ve increased our engagement with you through direct conversations and expanded our beta program, ensuring your voice is heard and valued.
- Monthly updates from Engineering leadership: Storyline’s engineering leader, Jesse Taber, provided monthly “State of Storyline” updates as comments on my original E-Learning Heroes post. In Jesse’s final update in September 2023 after we took Storyline out of Code Red, he committed to continue providing updates on a quarterly basis.
-
- The first quarterly Storyline quality update is now available.
As a bonus, Jesse also wrote a separate article all about 64-bit Storyline.
- The first quarterly Storyline quality update is now available.
- Commitment to working directly with you: We spoke with a lot of customers last year to learn about their experiences using Storyline 360 and have made that a regular practice. The insights from those conversations are invaluable. We also expanded participation in our private beta program by 55%! We love working directly with our private beta customers to gather feedback about new features and hear about issues. If you’re interested in a more direct line to our engineering team, please e-mail beta@articulate.com.
Looking Forward
As we move forward, balancing innovation with quality remains our unwavering commitment. Vanessa Fage, leading our Storyline quality team and beta program, is a testament to our dedication to continuous improvement and open communication.
A Year of Collaboration
This journey has reinforced the value of working directly with you, a lesson we'll carry forward to ensure we never stop listening again. Your trust fuels our progress, and for that, we’re profoundly grateful.
Thank you for inspiring us and for being a pivotal part of Storyline's evolution.
This month I’d like to talk about adoption rate and give a brief update on the other quality metrics we’ve been tracking.
Adoption Rate
Adoption rate refers to the rate at which Storyline 360 customers update to the latest version of the application. Storyline 360 updates are usually released every 4-6 weeks and customers can choose to install that update right away, wait awhile, or skip it entirely. We track the adoption rate for each new release by looking at telemetry data to determine what percentage of active users are running the latest version of the application in the first few weeks following its release.
The following table summarizes the adoption rate for all updates released in 2023 so far. Each row in this table represents a Storyline 360 update and the percentage of active users that were using that version in the 8 weeks following its initial release. The week that represents peak adoption of each update is bolded.
|
Update Number |
Release Date |
Week 1 |
Week 2 |
Week 3 |
Week 4 |
Week 5 |
Week 6 |
Week 7 |
Week 8 |
|
72 |
January 24, 2023 |
22.7% |
38.1% |
47.3% |
53.2% |
44.1% |
28.2% |
21% |
16.8% |
|
73 |
February 21, 2023 |
22.8% |
38.3% |
46.8% |
52.7% |
57.5% |
60.6% |
50.8% |
33.9% |
|
74 |
March 21, 2023 |
21.2% |
37.1% |
33.6% |
18.4% |
13% |
10.2% |
7.9% |
6.4% |
|
75 |
April 4, 2023 |
22.6% |
38.2% |
46.4% |
52% |
56.3% |
59.8% |
47.9% |
32.7% |
|
76 |
May 30, 2023 |
22.3% |
37.2% |
46.5% |
39.6% |
24.2% |
17.8% |
14.1% |
11.2% |
|
77 |
June 20, 2023 |
21.1% |
36.7% |
45.1% |
50.5% |
42.4% |
24.5% |
20.2% |
15.8% |
|
78* |
July 18, 2023 |
22.2% |
36.8% |
44.2% |
50.8% |
54.2% |
- |
- |
- |
*Update 78 was released 5 weeks ago so we don’t yet have numbers for weeks 6 - 8.
There are several facets of this data that I want to highlight:
- The percentage of users on the latest update in its first 4 weeks post-release is consistent:
- Week 1: ~21-22%
- Week 2: ~36-38%
- Week 3: ~44-48%
- Week 4: ~50-53%
- Each Storyline 360 update in 2023 reached its peak adoption in the last week in which it was the latest release. Once a new update comes out, it’s installed by ~21-22% of users in the first week and the previous update starts declining in usage.
- Updates 74 and 76 deviated from the adoption pattern in the first 4 weeks, but that is due to their relatively short tenure as the newest Storyline 360 release.
Ultimately our customers are free to choose when they want to upgrade to a new version of Storyline 360, though we recommend using the newest version to get the benefit of the latest features and bug fixes. We will continue to track the adoption rates of new releases to ensure that they don’t fall below the predictable pattern outlined above.
Other Notable Metrics
Here are updated numbers for application error rate, customer reported issues, downgrades, and defect rate:
Application Error Rate
Application error rate is a measure of how often Storyline displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions and users. Our goal is to drive both of these metrics under 1%. By driving this number down we’re helping ensure that Storyline 360 remains stable and responsive for our customers as they build courses.
|
Update Number |
Release Date |
Session Rate |
User Rate |
|
76 |
May 30, 2023 |
2.6% |
23.6% |
|
77 |
June 20, 2023 |
1.7% |
21% |
|
78 |
July 18, 2023 |
1.6% |
17.1% |
Customer Reported Issues
This is a breakdown of the bugs we’ve fixed and the number of customers that were affected by them for the past few releases.
|
Update Number |
Release Date |
Bugs Fixed |
Affected Customers |
|
76 |
May 30, 2023 |
33 |
506 |
|
77 |
June 20, 2023 |
23 |
312 |
|
78 |
July 18, 2023 |
11 |
197 |
Downgrades
This is the percentage of Storyline 360 users that install a given update only to later downgrade to an earlier version. If our customers have to downgrade that is a signal that their work is being impeded by issues introduced in a new version of the software. Our goal is to keep this metric under 1%.
|
Update Number |
Release Date |
Downgrade % |
|
76 |
May 30, 2023 |
0.9% |
|
77 |
June 20, 2023 |
0.9% |
|
78 |
July 18, 2023 |
0.76% |
Defect Rate
This is the percentage of open support cases that are associated with an unfixed bug. If this metric rises it’s an indication that our customers are experiencing pain due to bugs that need to be addressed. Our goal is to keep this metric under 12%.
|
Quarter |
Defect Rate |
|
Q1 2023 |
9.56% |
|
Q2 2023 |
6.62% |
|
Q3 2023* |
6.89% |
* The Q3 average is quarter-to-date
Wrap Up
To close out this month’s update I’d like to share a couple of additional items:
- Sign-in Issues: We are investigating customer reports of issues getting signed into the Articulate 360 Desktop application. If you have experienced issues signing in recently we’d like to invite you to fill out a brief survey about the issues you’ve experienced to help us better understand what is happening.
- Publishing Issues: We have noticed an uptick in the number of publish operations in Update 78 that appear to be failing. If you are experiencing instability or other issues when previewing or publishing in Update 78 of Storyline 360 please contact our Support team here.
- 64-bit Storyline 360: We remain committed to shipping a 64-bit version of Storyline 360. We are currently formulating plans to validate the stability of the 64-bit version of Storyline 360. Once that is complete we’ll share details about our plans to roll this out for customers to use. I’ll share some of these details in next month’s update.
- Storyline 360 Private Beta: I’d like to close out this monthly update by extending an invitation to join our private beta program. The beta version of Storyline 360 is updated weekly with new features and bug fixes before the public version. Participants in this program can report bugs or provide feedback directly to the Storyline 360 engineering team. The beta version can also be installed side-by-side with the public one so you don't have to choose one or the other. If you’re interested in participating please e-mail beta@articulate.com and we’ll get you added to the program!
