Forum Discussion
Storyline under a Microscope: An update from the CTO
Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where thing...
A Year of Change: Reflecting on 12 months of Storyline development
Since stepping into the CTO role at Articulate in early 2023, I've had the privilege of diving deep into our customer experiences. Back then, it quickly became clear that while there was a lot of love for Storyline, there were valid frustrations around unresolved bugs, lack of transparency, missing 64-bit support, and insufficient communication from our engineering team.
Hearing you loud and clear
About a year ago, your feedback was unmistakable: we needed to listen better. I shared our engineering teams’ commitment to change in a post titled, “Storyline under a Microscope: An update from the CTO”, and I'm thrilled to talk about the strides we've made since.
Our Shift in Focus
Today, our conversations with customers are future-focused, revolving around quality, advanced capabilities, and accessibility. It’s a shift from frustration to anticipation, and while we haven’t solved every issue, the progress is undeniable. So let’s review the key themes from the past year and highlight the progress we made on the journey.
Reflecting on Key Themes: Quality, Transparency, and Communication
Quality became focus number one for our team. In a nutshell, we hit pause on all new features to zero in on fixing existing issues, aligning our teams more closely for better quality control, diving into real customer scenarios to guide our releases, and doubling down on modernizing Storyline, including launching a 64-bit version in beta. In more detail:
- Prioritizing Quality Over Features: We took a strategic pause on developing new features for Storyline for six months, dedicating this period solely to enhancing product quality. The entire Storyline Engineering Team shifted their focus to addressing and resolving existing quality issues, ensuring a stronger foundation for our product.
- Unifying our Approach for Better Results: Previously, our Engineering, Quality Assurance, and Support teams worked independently on Storyline releases, which sometimes led to siloed efforts. We've since revamped our approach, bringing these teams into a cohesive unit. This collaboration ensures we can identify and address quality issues more efficiently, resulting in faster and more effective solutions.
- Learning Directly From User Experiences: In partnership with David Anderson, our Director of Customer Training, we've integrated real-world course-building scenarios into our testing process. These scenarios, reflective of our customers' daily challenges, are now a crucial part of validating each Storyline update before it goes live. This practice has not only improved our issue detection but has also deepened our understanding of what our users truly need.
- Modernizing Storyline: A year ago, we reinforced our commitment to Storyline by launching the first 64-bit version, Storyline 360 x64, in a public beta. We're now in the final stages of preparing Storyline 360 x64 for widespread release and are actively working on additional modernization projects to ensure Storyline continues to meet the evolving needs of our users.
Along with our investment in quality, we recognized the need to increase transparency to clarify how we make decisions around what we work on.
Commitment to Transparency
Starting last year we committed to being more transparent about how we approach building and maintaining Storyline. We opened up about our decision-making process, our progress toward reducing unexpected errors, and our plans for addressing long-standing bugs. Our revamped bug triage process ensures we're responsive and focused on what matters most.
- Reducing the error rate: When we started this effort in April of 2023, about 3% of Storyline sessions were encountering an unexpected error. We committed to getting this metric under 1%. As of March 1, 2024 1.2% of Storyline sessions are encountering an unexpected error. We’ve worked hard to reduce these errors and we won’t stop until we get under our 1% goal because any error that interrupts your day is like a “paper cut” in your way.
- Staying “below five”: Twelve months ago, we discovered we had stopped paying attention to “older bugs”. So we focused our attention on not just recent issues but all bugs that had five customers or more. We have 100s of thousands of users, and set a line in the sand that all bugs must stay “below five customers” otherwise we stop feature development to fix. We cleaned up our backlog of bugs affecting five or more customers and Storyline is much stronger for it.
- Revamping our bug triage process: Our bug database was, admittedly, a bit of a mess in April of 2023 and it was difficult to find signal in the noise. The right things were not always getting prioritized. So along with jumping on those older bugs, we revamped our daily bug triage approach to prioritize new bugs in the latest releases. We also started tracking the overall defect rate coming in from support cases to drive it below 10%. These changes helped us jump on issues causing pain quickly so that it didn’t spread.
To reinforce our quality and transparency commitment, we recognized the need to increase communications to ensure your concerns were being heard and addressed.
Emphasis on continuous communication
In summary, monthly updates last year have kept you informed. We’ve increased our engagement with you through direct conversations and expanded our beta program, ensuring your voice is heard and valued.
- Monthly updates from Engineering leadership: Storyline’s engineering leader, Jesse Taber, provided monthly “State of Storyline” updates as comments on my original E-Learning Heroes post. In Jesse’s final update in September 2023 after we took Storyline out of Code Red, he committed to continue providing updates on a quarterly basis.
-
- The first quarterly Storyline quality update is now available.
As a bonus, Jesse also wrote a separate article all about 64-bit Storyline.
- The first quarterly Storyline quality update is now available.
- Commitment to working directly with you: We spoke with a lot of customers last year to learn about their experiences using Storyline 360 and have made that a regular practice. The insights from those conversations are invaluable. We also expanded participation in our private beta program by 55%! We love working directly with our private beta customers to gather feedback about new features and hear about issues. If you’re interested in a more direct line to our engineering team, please e-mail beta@articulate.com.
Looking Forward
As we move forward, balancing innovation with quality remains our unwavering commitment. Vanessa Fage, leading our Storyline quality team and beta program, is a testament to our dedication to continuous improvement and open communication.
A Year of Collaboration
This journey has reinforced the value of working directly with you, a lesson we'll carry forward to ensure we never stop listening again. Your trust fuels our progress, and for that, we’re profoundly grateful.
Thank you for inspiring us and for being a pivotal part of Storyline's evolution.
Welcome to the September State of Storyline update. It’s been 6 months since we refocused the entire Storyline 360 team on improving quality and I’d like to summarize the progress we’ve made, what we've learned, and share our plans for continuing to improve our customers’ Storyline experience. Also, even more importantly, I’d like to provide an update on 64-bit Storyline including a proposed timeline for rolling it out to customers.
Quality Metrics
When we started focusing on quality in Storyline we defined several metrics to help us objectively measure our progress. We track these closely with each new Storyline release to see if we’re making progress or if there are areas that require our attention. I’ve discussed some of these metrics in prior updates, and this month I want to share a more comprehensive look at how they have improved since the start of 2023.
Application Error Rate
Application error rate is a measure of how often Storyline displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions and users.
When we started tracking this metric close to 3% of Storyline sessions experienced the error dialog at least once. We understand that seeing the error dialog is not only disruptive to your work but also reduces confidence in the stability of the application, so improving this metric has been our top priority. We started by addressing a relatively small number of extremely high-frequency errors to drive this metric down as quickly as possible. The chart indicates that most progress was made between the April and June releases and has slowed in recent weeks.
The recent slowdown is due to the fact that we have addressed all of the extremely high-frequency errors and are now focused on a larger number of lower-frequency errors. We are currently seeing an application error rate of 1.6% in the August update. We remain committed to addressing these errors until this rate is under 1% to provide a smooth experience for all of our customers.
Line chart showing Storyline 360’s application error rate per session for updates 73 through 79.
We initially found that over 30% of Storyline 360 users were seeing the error dialog in a given release. We’ve been able to drive this number down to 15% in the August update and will continue working toward our goal of getting this under 1% to ensure your work is not disrupted by errors.
Line chart showing Storyline 360’s application error rate per user for updates 73 through 79.
Downgrades
When we started tracking install, upgrade, and downgrade activity we found that roughly 2% of upgrades/installs would later downgrade to a previous version of Storyline. Updating to the latest version of Storyline 360 takes time and disrupts your workflow; having to later downgrade takes even more time that we know you’d rather spend building your courses. Our effort to improve this metric has been two-fold:
- Improve our pre-release quality assurance to find issues before they ship to customers.
- Closely monitor application health following a release to fix issues before a majority of customers have adopted the new version.
These efforts have reduced downgrades to 0.76%. We will continue to monitor this metric to ensure it stays under 1%.
Line chart showing Storyline 360’s downgrade percentage for updates 73 through 79.
Defect Rate
The defect rate metric tracks the percentage of open support cases that are associated with an unfixed bug. If this value spikes it’s usually a signal that we have bugs that are impacting many customers and need to be addressed. We capture this metric on a weekly basis and calculate rolled up monthly and quarterly averages. Our initial goal was to keep this value under 12%, but it’s improved so much recently that we’ve lowered that threshold to 10%. The monthly averages show a gradual trend downward over the course of this year with the month-to-date average for September currently at 4.7%. We will continue to monitor this and address any bugs that drive it over the 10% threshold to ensure that we’re addressing the issues having the broadest impact.
I want to emphasize the importance of reporting bugs to our Support team. Your reports help us prioritize bugs to address the ones impacting the most customers. If you’re experiencing an issue with Storyline 360 please get in touch with our Support team.
Line chart showing Storyline 360’s monthly defect rate percentage for 2023 year-to-date.
Publishing Failures
I have not shared this metric in previous updates, but we have been tracking the percentage of publish operations in Storyline 360 that fail due to an error. Publishing is probably the most important function in Storyline 360; no one can benefit from what you’ve built if you can’t publish it to share with the world. When we first started tracking this close to 6% of publish operations were failing. This value came down as we improved the overall application error rate, but then began to climb again. We are still investigating what is driving this increase, so if you are experiencing instability or other issues when previewing or publishing in Storyline 360 please contact our Support team. Our goal is to drive this number under 1% to ensure that you can get your courses in front of learners.
Line chart showing Storyline 360’s failed publish percentage for updates 73 through 79.
Incomplete Sessions
Another metric that we haven’t discussed in these updates is incomplete sessions. When Storyline first starts up we track a “begin session” event in our telemetry data. Later, when the user closes Storyline we record a corresponding “end session” event. We’re able to correlate these together to see what percentage of Storyline sessions recorded a begin event with no matching end event. There are two reasons a begin event may not have a matching end event:
- Storyline is still open.
- Storyline closed in a way that prevented us from recording the end session event, potentially because the process crashed and forced-quit due to an unexpected error.
Our incomplete session metric has hovered at ~6% since we started tracking it. We need to know how many of these incomplete sessions are the result of Storyline crashing unexpectedly, but there are facets of this data that we do not fully understand yet. Update 80 of Storyline 360 includes improvements to how we capture and report on incomplete sessions that will help us get a more accurate picture of how many incomplete sessions are the result of an unexpected crash. Once we know how many of these incomplete sessions are truly the result of Storyline crashing we’ll establish a goal for this metric and begin working toward it. Addressing unexpected crashes that force-quit Storyline is a top priority for the team because we understand how frustrating and disruptive these crashes can be.
Line chart showing Storyline 360’s failed publish percentage for updates 73 through 79.
64-bit Storyline 360
While much of the team has been focused on addressing bugs and crashes in recent months, we’ve also been working hard on creating a 64-bit version of Storyline 360. In my July update I announced that we had reached the milestone of creating 64-bit builds of Storyline for internal testing and validation. Since then we have completed most of the work needed to distribute and install 64-bit versions of Storyline 360 via the 360 Desktop application. I’ve been using 64-bit Storyline 360 for the past several days and am impressed with how well it works.
We’ve defined the next three milestones for this project:
- Internal Validation
- Private Beta
- Public Beta
Internal Validation
Our engineers are putting together a plan to have our team and other internal stakeholders validate the quality of the 64-bit version of Storyline 360. This will be a large team effort to put the application through its paces, paying particular attention to any areas that had to be updated to support 64-bit architecture.
Once this plan is complete, the team will spend about a week executing on it and addressing any issues we find. After the validation effort we will determine if 64-bit Storyline has met the bar to be released to the private beta program.
Timeframe: Internal validation should be complete by the end of September.
Private Beta
Like all new features, 64-bit Storyline will be available in the private beta before it’s more widely released. Customers in the private beta program will be able to install the 64-bit version of Storyline 360 alongside both the 32-bit public production and private beta versions, meaning they can continue using those for critical work-in-progress while evaluating the 64-bit version at their own pace.
The majority of the most popular Storyline 360 features will be available in the private beta for customers to evaluate, with some notable exceptions.
We are planning to cut the following features from the 64-bit version of Storyline. The cost to port these features to 64-bit exceeds the benefit they offer to customers.
- Engage Import
- Quizmaker ‘09 and ‘13 Import
- Flash Support
- Record Webcam
Additionally, two Storyline components are still being updated to support 64-bit architecture and will not be available when this lands in private beta. These features both rely on software components that only work in a 32-bit architecture. We are evaluating options for replacing or updating them.
- Screen Recording
- PowerPoint Import
We will closely monitor the private beta of 64-bit Storyline for reports of bugs or other issues to get them addressed as quickly as possible. Once we feel that it’s stable, we’ll release it to public beta.
Timeframe: When we release 64-bit to private beta will depend on what is found during internal validation, but we are tentatively planning on shipping to private beta in mid October. If you want to be among the first to try it out and are not yet part of our private beta program please e-mail beta@articulate.com to request access!
Public Beta
Our public beta program rolled out earlier this year. We think it’s important to let your feedback drive our efforts, and this beta program lets us get new features into customer hands sooner so your voice can be heard while development of the feature is still on-going.
Once the 64-bit version of Storyline is in public beta, all 360 subscribers will be able to evaluate it. Just like the private beta, you’ll be able to install the 64-bit version alongside the 32-bit one and evaluate it at your own pace.
Timeframe: Assuming we do not see major issues in the private beta, we hope to make the public beta of 64-bit Storyline available to all Articulate 360 subscribers before the end of 2023.
Wrap-Up
I have been a professional software engineer for over 2 decades, about half of which has been spent at Articulate. I can say, without hesitation, that these last 6 months have been some of the most challenging yet rewarding of my career. I am very proud of the progress this team has made, but also recognize that we have a lot left to do. I assure you that we do not take these quality issues lightly and that we are committed to making Storyline 360 the best it can be.
Our team remains fully committed to transparency and we will continue to share regular updates on our progress. However, we want to adjust the cadence and format to provide the most value. Going forward we plan to turn this into a quarterly article series on E-Learning Heroes. I will post a comment in this thread when the first quarterly article goes live, so please hit the “Subscribe” button at the top of this post to be sure you get notified.
If you have suggestions for topics to cover in future quarterly updates, any feedback about these updates or Storyline in general please reach out: jtaber@articulate.com.
