Forum Discussion
Storyline under a Microscope: An update from the CTO
Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where things stand and where they need to be.
There were a lot of positive signs that we already knew. For example, nearly a million courses were created with Storyline last year. But to be honest, we found things we didn’t like and were dismayed our heroes had to call us out to look deeper.
As an example of what we found, our error reporting shows us that 3% of application sessions hit an error. Based on some quick math, those errors add up to 3,000+ hours last year of course creator time spent facing errors. That was one example that surprised us. The other was “the state of defects” including the age of bugs, the number of outstanding defects reported by 5 or more customers, and the number of “bug workarounds” the community and our eLearning Heroes team come up with everyday.
As an engineering team, we’ve been very focused on new features, on new releases, and we relied on our QA and support partners to work directly with you all to keep us paying attention to the right things and keep our quality on track. But by being separated from our customers, we were missing the full picture.
We are committed to do the work to earn back your trust. We recognize that in our focus to build new capabilities, we took our eye off the foundations that matter to your success every day.
So here are the top three things we learned and what we plan to do about it.
Quality First: We must prioritize Storyline quality over everything else.
- First, we commit to prioritize quality over features. We recognize more features don’t help you succeed when you’re encountering paper cuts on a regular basis. Quality comes first. We’ve still got exciting new features on the horizon but we are doubling down on quality first.
- Second, we're changing our test and release cycle. The team operates on a monthly release cycle, with the final week focused on validation. We release a beta build to a dedicated group of customers on a weekly basis so they can test and report any issues to our team. Our Engineering, QA, and Support teams have previously operated in separate yet parallel tracks, but we’ve moved to a partnership where they can lean on each other to learn, test, and focus on customer needs. The next few releases will focus on quality and deepening these partnerships to create a better Storyline.
- Third, we will source validation scenarios from the community. We heard from a number of you who say you wait months before upgrading Storyline and do extensive validation in advance. We will source validation scenarios from the community to include directly in our testing.
- Fourth, we will invest in Storyline’s foundations. We are doing a deep dive into 64-bit support for Storyline and will report back with a plan. We had taken the stance that adding more capabilities to Storyline was job #1. So we were focused on delivering Storyline updates, along with quality updates, every month instead of disappearing into a cave on a long infrastructure project where the value of the impact is hard to measure until it is done. We are committed to investing in Storyline’s foundations so expect to hear more soon.
Transparency: We need to be more transparent about how we are prioritizing our work.
- First priority is to reduce the error rate to below 1%. You can’t build courses, if you keep running into errors. In release 74, we fixed the top 18% of those errors that generated support cases but there is a lot more work to be done so those errors don’t escape our tests. We’ll report back monthly to update how we are doing on this critical metric.
- Second we prioritized 35 older issues impacting five or more customers. We reviewed nearly 300 issues open over the life of Storyline. 35 of them are impacting five or more customers that were not at the top of our list. We are prioritizing those now and shipped a number of fixes for those in Release 74. Previously we focused our calories almost exclusively on more recent issues. So we’ve dedicated more engineers to focus on quality and renewed our focus on fixing older issues. As we work through those issues, we will share details.
- Third, we are cleaning up our bug database. As a bug ages, finding the issue and fixing it without impacting something else becomes increasingly complex. When we reviewed all open issues, we found issues over 3 years old that affected one customer and had a workaround. We aren’t going to fix those issues. All defects are a problem and we don’t mean to imply we don’t care about quality. We need to focus our team on what impacts the community the most, and report back to those customers transparently about what we won’t fix instead of pretending we might some day.
Communication: We need to communicate proactively instead of reactively.
- First, we will commit to monthly updates to the community from our Engineering team. We are working on the format and content to make it valuable, but to start we will commit to providing a “State of Storyline” monthly so you have the opportunity to know and react to what we are focused on. Quality is our number one priority right now and we are looking for measurable improvements that make a difference to you.
- Second, our engineering team and I commit to working directly with all of you. Our engineers and engineering leaders have kept a distance from you all by accident. We’ve been focused on building and relied on our partners inside Articulate to be the “voice of our customers” rather than listening directly. So this is a commitment for us to work better together directly. We appreciate the time, energy, and passion you all bring to Storyline and we commit to make that worth it.
Storyline 360 is our flagship product and is how you help your employees, your customers, and your companies to learn and grow everyday. We have a lot of exciting things planned this year but agree that quality must come first. If you’re interested in joining our beta program or sharing your teams’ Storyline validation scenarios, please email beta@articulate.com.
Our team will be updating this thread directly each month to keep you aware of our progress. To share feedback on our process, our plans, and our areas of focus please open a new thread or reach out to the support team so they can connect us directly.
We are committed to more transparency and proactive communication directly with the engineering team because honestly we are only here because of you.
A Year of Change: Reflecting on 12 months of Storyline development
Since stepping into the CTO role at Articulate in early 2023, I've had the privilege of diving deep into our customer experiences. Back then, it quickly became clear that while there was a lot of love for Storyline, there were valid frustrations around unresolved bugs, lack of transparency, missing 64-bit support, and insufficient communication from our engineering team.
Hearing you loud and clear
About a year ago, your feedback was unmistakable: we needed to listen better. I shared our engineering teams’ commitment to change in a post titled, “Storyline under a Microscope: An update from the CTO”, and I'm thrilled to talk about the strides we've made since.
Our Shift in Focus
Today, our conversations with customers are future-focused, revolving around quality, advanced capabilities, and accessibility. It’s a shift from frustration to anticipation, and while we haven’t solved every issue, the progress is undeniable. So let’s review the key themes from the past year and highlight the progress we made on the journey.
Reflecting on Key Themes: Quality, Transparency, and Communication
Quality became focus number one for our team. In a nutshell, we hit pause on all new features to zero in on fixing existing issues, aligning our teams more closely for better quality control, diving into real customer scenarios to guide our releases, and doubling down on modernizing Storyline, including launching a 64-bit version in beta. In more detail:
- Prioritizing Quality Over Features: We took a strategic pause on developing new features for Storyline for six months, dedicating this period solely to enhancing product quality. The entire Storyline Engineering Team shifted their focus to addressing and resolving existing quality issues, ensuring a stronger foundation for our product.
- Unifying our Approach for Better Results: Previously, our Engineering, Quality Assurance, and Support teams worked independently on Storyline releases, which sometimes led to siloed efforts. We've since revamped our approach, bringing these teams into a cohesive unit. This collaboration ensures we can identify and address quality issues more efficiently, resulting in faster and more effective solutions.
- Learning Directly From User Experiences: In partnership with David Anderson, our Director of Customer Training, we've integrated real-world course-building scenarios into our testing process. These scenarios, reflective of our customers' daily challenges, are now a crucial part of validating each Storyline update before it goes live. This practice has not only improved our issue detection but has also deepened our understanding of what our users truly need.
- Modernizing Storyline: A year ago, we reinforced our commitment to Storyline by launching the first 64-bit version, Storyline 360 x64, in a public beta. We're now in the final stages of preparing Storyline 360 x64 for widespread release and are actively working on additional modernization projects to ensure Storyline continues to meet the evolving needs of our users.
Along with our investment in quality, we recognized the need to increase transparency to clarify how we make decisions around what we work on.
Commitment to Transparency
Starting last year we committed to being more transparent about how we approach building and maintaining Storyline. We opened up about our decision-making process, our progress toward reducing unexpected errors, and our plans for addressing long-standing bugs. Our revamped bug triage process ensures we're responsive and focused on what matters most.
- Reducing the error rate: When we started this effort in April of 2023, about 3% of Storyline sessions were encountering an unexpected error. We committed to getting this metric under 1%. As of March 1, 2024 1.2% of Storyline sessions are encountering an unexpected error. We’ve worked hard to reduce these errors and we won’t stop until we get under our 1% goal because any error that interrupts your day is like a “paper cut” in your way.
- Staying “below five”: Twelve months ago, we discovered we had stopped paying attention to “older bugs”. So we focused our attention on not just recent issues but all bugs that had five customers or more. We have 100s of thousands of users, and set a line in the sand that all bugs must stay “below five customers” otherwise we stop feature development to fix. We cleaned up our backlog of bugs affecting five or more customers and Storyline is much stronger for it.
- Revamping our bug triage process: Our bug database was, admittedly, a bit of a mess in April of 2023 and it was difficult to find signal in the noise. The right things were not always getting prioritized. So along with jumping on those older bugs, we revamped our daily bug triage approach to prioritize new bugs in the latest releases. We also started tracking the overall defect rate coming in from support cases to drive it below 10%. These changes helped us jump on issues causing pain quickly so that it didn’t spread.
To reinforce our quality and transparency commitment, we recognized the need to increase communications to ensure your concerns were being heard and addressed.
Emphasis on continuous communication
In summary, monthly updates last year have kept you informed. We’ve increased our engagement with you through direct conversations and expanded our beta program, ensuring your voice is heard and valued.
- Monthly updates from Engineering leadership: Storyline’s engineering leader, Jesse Taber, provided monthly “State of Storyline” updates as comments on my original E-Learning Heroes post. In Jesse’s final update in September 2023 after we took Storyline out of Code Red, he committed to continue providing updates on a quarterly basis.
-
- The first quarterly Storyline quality update is now available.
As a bonus, Jesse also wrote a separate article all about 64-bit Storyline.
- The first quarterly Storyline quality update is now available.
- Commitment to working directly with you: We spoke with a lot of customers last year to learn about their experiences using Storyline 360 and have made that a regular practice. The insights from those conversations are invaluable. We also expanded participation in our private beta program by 55%! We love working directly with our private beta customers to gather feedback about new features and hear about issues. If you’re interested in a more direct line to our engineering team, please e-mail beta@articulate.com.
Looking Forward
As we move forward, balancing innovation with quality remains our unwavering commitment. Vanessa Fage, leading our Storyline quality team and beta program, is a testament to our dedication to continuous improvement and open communication.
A Year of Collaboration
This journey has reinforced the value of working directly with you, a lesson we'll carry forward to ensure we never stop listening again. Your trust fuels our progress, and for that, we’re profoundly grateful.
Thank you for inspiring us and for being a pivotal part of Storyline's evolution.
Last month we shared our plans to address quality issues in Storyline 360 and committed to monthly updates in the interest of open, honest, and transparent communication with our customers.
My name is Jesse Taber, and I’m the Engineering Manager for the Storyline 360 Team. I’d like to share our progress in 3 key areas related to improving Storyline 360’s overall quality and stability: application error rate, customer-reported issues, and pre-release quality assurance.
Application Error Rate
You’re working hard on your course, you make a seemingly innocuous change, and suddenly you’re confronted with the “Articulate Storyline Error Report” dialog or even worse, the Storyline application crashes completely and has to be restarted. Not only are these errors an impediment to your work, but they shake your confidence in the application. For these reasons, our primary focus over the past month has been investigating and resolving these errors.
Last month Kerry mentioned that ~3% of Storyline sessions were encountering an error at least once. During the last 3 releases of Storyline 360 ~30% of users were encountering the error dialog at least once per month. Our goal is to drive both of these metrics down under 1%.
In Update 75 of Storyline 360, we addressed 9 of the most common errors that users were experiencing during their Storyline sessions. These errors affected a wide range of Storyline 360 functionality, including manipulating shape states, interacting with the ribbon, and editing or publishing videos. Update 75 was released on April 18, and we are seeing the following improvements to our average error rate metrics:
% of application sessions encountering the error dialog at least once: 2.7% (~0.3% improvement)
% of users encountering the error dialog at least once per month: 22.8% (~7.2% improvement)
While these metrics are promising, we recognize that we have a lot more work to do. We are devoting engineering resources to address these errors until we reach our goal.
On a related note: The Articulate Storyline Error Report dialog allows you to share additional information about the circumstances that led to the error. This goes directly to the Storyline 360 engineering team to use when investigating errors. It does not create a Support case for follow-up, but you can always connect with our Support team here.
Customer Reported Issues
We have been working to clean up our bug database. Our Engineering and Customer Support teams have been working closely together to dive deep on customer-reported issues to be sure that we have the most accurate picture possible of the issues that are impacting the most customers. This effort has led to many issues being de-duplicated and consolidated so that we have a much clearer idea of where to focus our bug-fixing efforts. Prior to this work we had identified 35 issues impacting 5 or more customer accounts based on support case data, but that number rose to 44 as a result of our de-duplication and consolidation.
In Update 75 of Storyline 360 we addressed 5 of these issues affecting 5 or more customers based on support case data, bringing that count down to 39. These issues impacted areas such as video compression, sliders, and performance in the Storyline 360 HTML5 player application.
Additionally, we addressed 5 issues that were reported and discussed by users in public forums such as Twitter or our own E-Learning Heroes forum. These issues impacted areas such as the focus order of objects on a slide, the “print results” action for quizzes, and an annoying bug where the Storyline 360 HTML5 Player would show the Resources tab even if the author had unchecked that option.
Pre-release Quality Assurance
We shared that we were making changes to our test and release cycles with an eye for improving the techniques used to validate quality as well as promote better alignment between the Engineering and Support teams. This effort will be continuously refined, but we have taken the following measures to start:
- Establishing a formal Support Liaison role: The Support Liaison role was created to help bridge the gap between Engineering and Support and promote better cooperation between these teams. The Support Liaison works very closely with the Storyline 360 engineering team to ensure that we are hearing the voice of the customer. This role will share customer feedback with us (good, bad, and ugly), bring issues to our attention that might not yet have surfaced through our normal support case data analysis, and bring the support team's perspective on what issues the team should focus on next. The entire engineering team is focused on getting closer to our customers, and the Support Liaison is a big step in that direction.
- Sourcing validation scenarios: Storyline 360 is a large, complex application with many features. We know that our customers use it to create interactive, entertaining, informative, and beautiful courses every day, but when we’re heads-down buried in the minutiae of the code on a daily basis, it can be easy to lose sight of the impressive ways our customers use the application every day. When we’re validating a new Storyline 360 release we try to use the product similarly to how our customers do. We’ve tapped into our very talented training team and asked them to define the common usage scenarios in Storyline. We now have a suite of common Storyline 360 course scenarios, such as conditional navigation, tabs interactions, and software simulations, that we use to put the application through its paces prior to releasing it publicly. We’ll continue to look for new validation scenarios from both the customer community and our own Storyline 360 experts to ensure we’re testing things in the right way.
Looking Forward
To wrap up this month’s update, I’d like to share things we’re actively working on:
- 64-bit Storyline 360: We’re doing a deep dive on what it will take to create a 64-bit version of Storyline 360. Storyline is a large and complex piece of software comprised of many different components, both created internally by Articulate and sourced from third parties. Some of these components were not designed to work outside of a 32-bit process architecture and will need to be updated, replaced, or removed. We have identified the components that represent the largest amounts of effort and highest risk to the success of a 64-bit conversion. Our next step is to evaluate our options on a per-component basis and determine the plan for each. We will begin that effort this month and will share updates on our progress.
- Storyline 360 Private Beta: Storyline 360 has a private beta program in which the software is updated weekly. The beta version gets new features and bug fixes before the public version does and serves as a great way for us to gather customer feedback or identify issues before the changes are made widely available. Any bug reports or feedback made through the private beta go directly to the Storyline 360 engineering team for review. Last month we extended an invitation for anyone interested in participating to e-mail beta@articulate.com, and we would like to remind everyone that this invitation still stands. Participating in the private beta helps make the product better and gives you an inside track on what changes are coming in future public releases.
Welcome to the September State of Storyline update. It’s been 6 months since we refocused the entire Storyline 360 team on improving quality and I’d like to summarize the progress we’ve made, what we've learned, and share our plans for continuing to improve our customers’ Storyline experience. Also, even more importantly, I’d like to provide an update on 64-bit Storyline including a proposed timeline for rolling it out to customers.
Quality Metrics
When we started focusing on quality in Storyline we defined several metrics to help us objectively measure our progress. We track these closely with each new Storyline release to see if we’re making progress or if there are areas that require our attention. I’ve discussed some of these metrics in prior updates, and this month I want to share a more comprehensive look at how they have improved since the start of 2023.
Application Error Rate
Application error rate is a measure of how often Storyline displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions and users.
When we started tracking this metric close to 3% of Storyline sessions experienced the error dialog at least once. We understand that seeing the error dialog is not only disruptive to your work but also reduces confidence in the stability of the application, so improving this metric has been our top priority. We started by addressing a relatively small number of extremely high-frequency errors to drive this metric down as quickly as possible. The chart indicates that most progress was made between the April and June releases and has slowed in recent weeks.
The recent slowdown is due to the fact that we have addressed all of the extremely high-frequency errors and are now focused on a larger number of lower-frequency errors. We are currently seeing an application error rate of 1.6% in the August update. We remain committed to addressing these errors until this rate is under 1% to provide a smooth experience for all of our customers.
Line chart showing Storyline 360’s application error rate per session for updates 73 through 79.
We initially found that over 30% of Storyline 360 users were seeing the error dialog in a given release. We’ve been able to drive this number down to 15% in the August update and will continue working toward our goal of getting this under 1% to ensure your work is not disrupted by errors.
Line chart showing Storyline 360’s application error rate per user for updates 73 through 79.
Downgrades
When we started tracking install, upgrade, and downgrade activity we found that roughly 2% of upgrades/installs would later downgrade to a previous version of Storyline. Updating to the latest version of Storyline 360 takes time and disrupts your workflow; having to later downgrade takes even more time that we know you’d rather spend building your courses. Our effort to improve this metric has been two-fold:
- Improve our pre-release quality assurance to find issues before they ship to customers.
- Closely monitor application health following a release to fix issues before a majority of customers have adopted the new version.
These efforts have reduced downgrades to 0.76%. We will continue to monitor this metric to ensure it stays under 1%.
Line chart showing Storyline 360’s downgrade percentage for updates 73 through 79.
Defect Rate
The defect rate metric tracks the percentage of open support cases that are associated with an unfixed bug. If this value spikes it’s usually a signal that we have bugs that are impacting many customers and need to be addressed. We capture this metric on a weekly basis and calculate rolled up monthly and quarterly averages. Our initial goal was to keep this value under 12%, but it’s improved so much recently that we’ve lowered that threshold to 10%. The monthly averages show a gradual trend downward over the course of this year with the month-to-date average for September currently at 4.7%. We will continue to monitor this and address any bugs that drive it over the 10% threshold to ensure that we’re addressing the issues having the broadest impact.
I want to emphasize the importance of reporting bugs to our Support team. Your reports help us prioritize bugs to address the ones impacting the most customers. If you’re experiencing an issue with Storyline 360 please get in touch with our Support team.
Line chart showing Storyline 360’s monthly defect rate percentage for 2023 year-to-date.
Publishing Failures
I have not shared this metric in previous updates, but we have been tracking the percentage of publish operations in Storyline 360 that fail due to an error. Publishing is probably the most important function in Storyline 360; no one can benefit from what you’ve built if you can’t publish it to share with the world. When we first started tracking this close to 6% of publish operations were failing. This value came down as we improved the overall application error rate, but then began to climb again. We are still investigating what is driving this increase, so if you are experiencing instability or other issues when previewing or publishing in Storyline 360 please contact our Support team. Our goal is to drive this number under 1% to ensure that you can get your courses in front of learners.
Line chart showing Storyline 360’s failed publish percentage for updates 73 through 79.
Incomplete Sessions
Another metric that we haven’t discussed in these updates is incomplete sessions. When Storyline first starts up we track a “begin session” event in our telemetry data. Later, when the user closes Storyline we record a corresponding “end session” event. We’re able to correlate these together to see what percentage of Storyline sessions recorded a begin event with no matching end event. There are two reasons a begin event may not have a matching end event:
- Storyline is still open.
- Storyline closed in a way that prevented us from recording the end session event, potentially because the process crashed and forced-quit due to an unexpected error.
Our incomplete session metric has hovered at ~6% since we started tracking it. We need to know how many of these incomplete sessions are the result of Storyline crashing unexpectedly, but there are facets of this data that we do not fully understand yet. Update 80 of Storyline 360 includes improvements to how we capture and report on incomplete sessions that will help us get a more accurate picture of how many incomplete sessions are the result of an unexpected crash. Once we know how many of these incomplete sessions are truly the result of Storyline crashing we’ll establish a goal for this metric and begin working toward it. Addressing unexpected crashes that force-quit Storyline is a top priority for the team because we understand how frustrating and disruptive these crashes can be.
Line chart showing Storyline 360’s failed publish percentage for updates 73 through 79.
64-bit Storyline 360
While much of the team has been focused on addressing bugs and crashes in recent months, we’ve also been working hard on creating a 64-bit version of Storyline 360. In my July update I announced that we had reached the milestone of creating 64-bit builds of Storyline for internal testing and validation. Since then we have completed most of the work needed to distribute and install 64-bit versions of Storyline 360 via the 360 Desktop application. I’ve been using 64-bit Storyline 360 for the past several days and am impressed with how well it works.
We’ve defined the next three milestones for this project:
- Internal Validation
- Private Beta
- Public Beta
Internal Validation
Our engineers are putting together a plan to have our team and other internal stakeholders validate the quality of the 64-bit version of Storyline 360. This will be a large team effort to put the application through its paces, paying particular attention to any areas that had to be updated to support 64-bit architecture.
Once this plan is complete, the team will spend about a week executing on it and addressing any issues we find. After the validation effort we will determine if 64-bit Storyline has met the bar to be released to the private beta program.
Timeframe: Internal validation should be complete by the end of September.
Private Beta
Like all new features, 64-bit Storyline will be available in the private beta before it’s more widely released. Customers in the private beta program will be able to install the 64-bit version of Storyline 360 alongside both the 32-bit public production and private beta versions, meaning they can continue using those for critical work-in-progress while evaluating the 64-bit version at their own pace.
The majority of the most popular Storyline 360 features will be available in the private beta for customers to evaluate, with some notable exceptions.
We are planning to cut the following features from the 64-bit version of Storyline. The cost to port these features to 64-bit exceeds the benefit they offer to customers.
- Engage Import
- Quizmaker ‘09 and ‘13 Import
- Flash Support
- Record Webcam
Additionally, two Storyline components are still being updated to support 64-bit architecture and will not be available when this lands in private beta. These features both rely on software components that only work in a 32-bit architecture. We are evaluating options for replacing or updating them.
- Screen Recording
- PowerPoint Import
We will closely monitor the private beta of 64-bit Storyline for reports of bugs or other issues to get them addressed as quickly as possible. Once we feel that it’s stable, we’ll release it to public beta.
Timeframe: When we release 64-bit to private beta will depend on what is found during internal validation, but we are tentatively planning on shipping to private beta in mid October. If you want to be among the first to try it out and are not yet part of our private beta program please e-mail beta@articulate.com to request access!
Public Beta
Our public beta program rolled out earlier this year. We think it’s important to let your feedback drive our efforts, and this beta program lets us get new features into customer hands sooner so your voice can be heard while development of the feature is still on-going.
Once the 64-bit version of Storyline is in public beta, all 360 subscribers will be able to evaluate it. Just like the private beta, you’ll be able to install the 64-bit version alongside the 32-bit one and evaluate it at your own pace.
Timeframe: Assuming we do not see major issues in the private beta, we hope to make the public beta of 64-bit Storyline available to all Articulate 360 subscribers before the end of 2023.
Wrap-Up
I have been a professional software engineer for over 2 decades, about half of which has been spent at Articulate. I can say, without hesitation, that these last 6 months have been some of the most challenging yet rewarding of my career. I am very proud of the progress this team has made, but also recognize that we have a lot left to do. I assure you that we do not take these quality issues lightly and that we are committed to making Storyline 360 the best it can be.
Our team remains fully committed to transparency and we will continue to share regular updates on our progress. However, we want to adjust the cadence and format to provide the most value. Going forward we plan to turn this into a quarterly article series on E-Learning Heroes. I will post a comment in this thread when the first quarterly article goes live, so please hit the “Subscribe” button at the top of this post to be sure you get notified.
If you have suggestions for topics to cover in future quarterly updates, any feedback about these updates or Storyline in general please reach out: jtaber@articulate.com.
Welcome to the July 2023 “State of Storyline” update. When we began to focus on quality earlier this year we defined some key metrics to measure the overall stability and quality of Storyline. This month I’d like to share details about some of those metrics and the progress we’ve made in improving them over the past several months
Application Error Rate
The application error rate metric is based on how often Storyline displays the “Articulate Storyline Error Report” dialog. We track and report on this data in two ways:
- Session Application Error Rate: % of Storyline sessions that encounter the error report dialog at least once per release
- User Application Error Rate: % of Storyline users that encounter the error report dialog at least once per release
In April of 2023, our application error rates were ~3% for Storyline sessions and ~30% for Storyline users. Several weeks after shipping the Update 75 release of Storyline we saw these numbers decline, but unfortunately, Microsoft released a Windows Update in late March that could cause intermittent issues when publishing Storyline projects. The rollout of this Windows Update coincided with the adoption of Update 75 which led to an inflated application error rate for that release. We added some additional error handling to mitigate this problem in Update 76, but the core issue was in the Windows operating system itself so we were unable to resolve it completely. Last month Microsoft released a fix for this issue, and we are now seeing improved error rate metrics. Here are the error rates for the past three Storyline releases:
Update Number
Release Date
Session Rate
User Rate
75
April 18, 2023
3%
31%
76
May 30, 2023
2.6%
22%
77
June 20, 2023
1.8%
14%
This trend is very encouraging and we will continue working to drive both of these error rates under 1% in future releases.
Customer Reported Issues
Earlier this year we established a goal of addressing all bugs that are impacting 5 or more customer accounts. Here is a breakdown of the bugs we’ve fixed and the number of customers that were affected by them over the past three releases:
Update Number
Release Date
Bugs Fixed
Affected Customers
75
April 18, 2023
16
200
76
May 30, 2023
33
506
77
June 20, 2023
23
312
- In the last 3 months, we fixed 72 bugs affecting over 1,000 customers.
- For contrast, in the first 3 months of the year, we fixed a total of 41 bugs affecting just over 500 customers.
We are currently tracking 21 bugs impacting 5 or more customers and will continue to prioritize fixing those along with others that are causing customer pain.
Downgrades
Last year we started tracking how often Storyline 360 users upgrade to a new version of the software and then later downgrade to an earlier version. Because the adoption rate for new Storyline 360 releases is very consistent, we use downgrade frequency as a way to measure stability.
We established a goal to maintain the number of downgrades for each Storyline 360 release at less than 1% of the total number of new installs and upgrades. Here is a breakdown of the downgrade percentages for each release in 2023:
Update Number
Release Date
Downgrade %
72
January 23, 2023
2.64%
73
February 21, 2023
1.99%
74
March 21, 2023
1.03%
75
April 4, 2023
0.94%
76
May 30, 2023
0.82%
77
June 20, 2023
0.77%
We interpret the improved downgrade percentage numbers to mean that we’re both shipping more stable software and are able to find and patch issues in new releases quickly. It’s been extremely rewarding for the team to see this metric improve as a result of our focus on quality. We’re very pleased to have reached our goal and will continue monitoring to ensure it remains at this level.
Defect Rate
Customers open support cases for many reasons. Sometimes they need help building a course, other times they may have billing questions, and sometimes they are reporting a bug in the software. The defect rate metric tracks the percentage of open support cases that are associated with an unfixed bug. The lower this number, the better. We snapshot this data on a weekly basis and roll it up into quarterly averages. Our current goal is to keep this metric under 12%.
In Q1 of 2023, our defect rate average was 9.56%. In Q2 we have been able to reduce it to 6.62%. The reduction in the defect rate is a direct result of our focus on both fixing existing bugs and improving our quality processes to reduce the number of new bugs being introduced.
Wrap Up
To close out this month’s update I’d like to share a couple of additional items:
- 64-bit Storyline 360: We recently reached an important milestone: for the first time ever we have 64-bit builds of Storyline 360 available for internal testing and validation! These builds have limited functionality and there is still much more work to be done, but the team has made great progress on this difficult task. We are continuing to update or replace 3rd party components that only work with 32-bit applications and are also updating parts of the Storyline infrastructure to ensure that we can correctly consume low-level Windows APIs in the 64-bit builds. Other remaining work includes creating an installer, updating our internal build and deploy infrastructure, and a lot more testing and validation.
- Storyline 360 Private Beta: I’d like to close out this monthly update by extending an invitation to join our private beta program. The beta version of Storyline 360 is updated weekly with new features and bug fixes before the public version. Participants in this program can report bugs or provide feedback directly to the Storyline 360 engineering team. The beta version can also be installed side-by-side with the public one so you don't have to choose one or the other. If you’re interested in participating please e-mail beta@articulate.com and we’ll get you added to the program!
A Year of Change: Reflecting on 12 months of Storyline development
Since stepping into the CTO role at Articulate in early 2023, I've had the privilege of diving deep into our customer experiences. Back then, it quickly became clear that while there was a lot of love for Storyline, there were valid frustrations around unresolved bugs, lack of transparency, missing 64-bit support, and insufficient communication from our engineering team.
Hearing you loud and clear
About a year ago, your feedback was unmistakable: we needed to listen better. I shared our engineering teams’ commitment to change in a post titled, “Storyline under a Microscope: An update from the CTO”, and I'm thrilled to talk about the strides we've made since.
Our Shift in Focus
Today, our conversations with customers are future-focused, revolving around quality, advanced capabilities, and accessibility. It’s a shift from frustration to anticipation, and while we haven’t solved every issue, the progress is undeniable. So let’s review the key themes from the past year and highlight the progress we made on the journey.
Reflecting on Key Themes: Quality, Transparency, and Communication
Quality became focus number one for our team. In a nutshell, we hit pause on all new features to zero in on fixing existing issues, aligning our teams more closely for better quality control, diving into real customer scenarios to guide our releases, and doubling down on modernizing Storyline, including launching a 64-bit version in beta. In more detail:
- Prioritizing Quality Over Features: We took a strategic pause on developing new features for Storyline for six months, dedicating this period solely to enhancing product quality. The entire Storyline Engineering Team shifted their focus to addressing and resolving existing quality issues, ensuring a stronger foundation for our product.
- Unifying our Approach for Better Results: Previously, our Engineering, Quality Assurance, and Support teams worked independently on Storyline releases, which sometimes led to siloed efforts. We've since revamped our approach, bringing these teams into a cohesive unit. This collaboration ensures we can identify and address quality issues more efficiently, resulting in faster and more effective solutions.
- Learning Directly From User Experiences: In partnership with David Anderson, our Director of Customer Training, we've integrated real-world course-building scenarios into our testing process. These scenarios, reflective of our customers' daily challenges, are now a crucial part of validating each Storyline update before it goes live. This practice has not only improved our issue detection but has also deepened our understanding of what our users truly need.
- Modernizing Storyline: A year ago, we reinforced our commitment to Storyline by launching the first 64-bit version, Storyline 360 x64, in a public beta. We're now in the final stages of preparing Storyline 360 x64 for widespread release and are actively working on additional modernization projects to ensure Storyline continues to meet the evolving needs of our users.
Along with our investment in quality, we recognized the need to increase transparency to clarify how we make decisions around what we work on.
Commitment to Transparency
Starting last year we committed to being more transparent about how we approach building and maintaining Storyline. We opened up about our decision-making process, our progress toward reducing unexpected errors, and our plans for addressing long-standing bugs. Our revamped bug triage process ensures we're responsive and focused on what matters most.
- Reducing the error rate: When we started this effort in April of 2023, about 3% of Storyline sessions were encountering an unexpected error. We committed to getting this metric under 1%. As of March 1, 2024 1.2% of Storyline sessions are encountering an unexpected error. We’ve worked hard to reduce these errors and we won’t stop until we get under our 1% goal because any error that interrupts your day is like a “paper cut” in your way.
- Staying “below five”: Twelve months ago, we discovered we had stopped paying attention to “older bugs”. So we focused our attention on not just recent issues but all bugs that had five customers or more. We have 100s of thousands of users, and set a line in the sand that all bugs must stay “below five customers” otherwise we stop feature development to fix. We cleaned up our backlog of bugs affecting five or more customers and Storyline is much stronger for it.
- Revamping our bug triage process: Our bug database was, admittedly, a bit of a mess in April of 2023 and it was difficult to find signal in the noise. The right things were not always getting prioritized. So along with jumping on those older bugs, we revamped our daily bug triage approach to prioritize new bugs in the latest releases. We also started tracking the overall defect rate coming in from support cases to drive it below 10%. These changes helped us jump on issues causing pain quickly so that it didn’t spread.
To reinforce our quality and transparency commitment, we recognized the need to increase communications to ensure your concerns were being heard and addressed.
Emphasis on continuous communication
In summary, monthly updates last year have kept you informed. We’ve increased our engagement with you through direct conversations and expanded our beta program, ensuring your voice is heard and valued.
- Monthly updates from Engineering leadership: Storyline’s engineering leader, Jesse Taber, provided monthly “State of Storyline” updates as comments on my original E-Learning Heroes post. In Jesse’s final update in September 2023 after we took Storyline out of Code Red, he committed to continue providing updates on a quarterly basis.
-
- The first quarterly Storyline quality update is now available.
As a bonus, Jesse also wrote a separate article all about 64-bit Storyline.
- The first quarterly Storyline quality update is now available.
- Commitment to working directly with you: We spoke with a lot of customers last year to learn about their experiences using Storyline 360 and have made that a regular practice. The insights from those conversations are invaluable. We also expanded participation in our private beta program by 55%! We love working directly with our private beta customers to gather feedback about new features and hear about issues. If you’re interested in a more direct line to our engineering team, please e-mail beta@articulate.com.
Looking Forward
As we move forward, balancing innovation with quality remains our unwavering commitment. Vanessa Fage, leading our Storyline quality team and beta program, is a testament to our dedication to continuous improvement and open communication.
A Year of Collaboration
This journey has reinforced the value of working directly with you, a lesson we'll carry forward to ensure we never stop listening again. Your trust fuels our progress, and for that, we’re profoundly grateful.
Thank you for inspiring us and for being a pivotal part of Storyline's evolution.
This month I’d like to talk about adoption rate and give a brief update on the other quality metrics we’ve been tracking.
Adoption Rate
Adoption rate refers to the rate at which Storyline 360 customers update to the latest version of the application. Storyline 360 updates are usually released every 4-6 weeks and customers can choose to install that update right away, wait awhile, or skip it entirely. We track the adoption rate for each new release by looking at telemetry data to determine what percentage of active users are running the latest version of the application in the first few weeks following its release.
The following table summarizes the adoption rate for all updates released in 2023 so far. Each row in this table represents a Storyline 360 update and the percentage of active users that were using that version in the 8 weeks following its initial release. The week that represents peak adoption of each update is bolded.
Update Number
Release Date
Week 1
Week 2
Week 3
Week 4
Week 5
Week 6
Week 7
Week 8
72
January 24, 2023
22.7%
38.1%
47.3%
53.2%
44.1%
28.2%
21%
16.8%
73
February 21, 2023
22.8%
38.3%
46.8%
52.7%
57.5%
60.6%
50.8%
33.9%
74
March 21, 2023
21.2%
37.1%
33.6%
18.4%
13%
10.2%
7.9%
6.4%
75
April 4, 2023
22.6%
38.2%
46.4%
52%
56.3%
59.8%
47.9%
32.7%
76
May 30, 2023
22.3%
37.2%
46.5%
39.6%
24.2%
17.8%
14.1%
11.2%
77
June 20, 2023
21.1%
36.7%
45.1%
50.5%
42.4%
24.5%
20.2%
15.8%
78*
July 18, 2023
22.2%
36.8%
44.2%
50.8%
54.2%
-
-
-
*Update 78 was released 5 weeks ago so we don’t yet have numbers for weeks 6 - 8.
There are several facets of this data that I want to highlight:
- The percentage of users on the latest update in its first 4 weeks post-release is consistent:
- Week 1: ~21-22%
- Week 2: ~36-38%
- Week 3: ~44-48%
- Week 4: ~50-53%
- Each Storyline 360 update in 2023 reached its peak adoption in the last week in which it was the latest release. Once a new update comes out, it’s installed by ~21-22% of users in the first week and the previous update starts declining in usage.
- Updates 74 and 76 deviated from the adoption pattern in the first 4 weeks, but that is due to their relatively short tenure as the newest Storyline 360 release.
Ultimately our customers are free to choose when they want to upgrade to a new version of Storyline 360, though we recommend using the newest version to get the benefit of the latest features and bug fixes. We will continue to track the adoption rates of new releases to ensure that they don’t fall below the predictable pattern outlined above.
Other Notable Metrics
Here are updated numbers for application error rate, customer reported issues, downgrades, and defect rate:
Application Error Rate
Application error rate is a measure of how often Storyline displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions and users. Our goal is to drive both of these metrics under 1%. By driving this number down we’re helping ensure that Storyline 360 remains stable and responsive for our customers as they build courses.
Update Number
Release Date
Session Rate
User Rate
76
May 30, 2023
2.6%
23.6%
77
June 20, 2023
1.7%
21%
78
July 18, 2023
1.6%
17.1%
Customer Reported Issues
This is a breakdown of the bugs we’ve fixed and the number of customers that were affected by them for the past few releases.
Update Number
Release Date
Bugs Fixed
Affected Customers
76
May 30, 2023
33
506
77
June 20, 2023
23
312
78
July 18, 2023
11
197
Downgrades
This is the percentage of Storyline 360 users that install a given update only to later downgrade to an earlier version. If our customers have to downgrade that is a signal that their work is being impeded by issues introduced in a new version of the software. Our goal is to keep this metric under 1%.
Update Number
Release Date
Downgrade %
76
May 30, 2023
0.9%
77
June 20, 2023
0.9%
78
July 18, 2023
0.76%
Defect Rate
This is the percentage of open support cases that are associated with an unfixed bug. If this metric rises it’s an indication that our customers are experiencing pain due to bugs that need to be addressed. Our goal is to keep this metric under 12%.
Quarter
Defect Rate
Q1 2023
9.56%
Q2 2023
6.62%
Q3 2023*
6.89%
* The Q3 average is quarter-to-date
Wrap Up
To close out this month’s update I’d like to share a couple of additional items:
- Sign-in Issues: We are investigating customer reports of issues getting signed into the Articulate 360 Desktop application. If you have experienced issues signing in recently we’d like to invite you to fill out a brief survey about the issues you’ve experienced to help us better understand what is happening.
- Publishing Issues: We have noticed an uptick in the number of publish operations in Update 78 that appear to be failing. If you are experiencing instability or other issues when previewing or publishing in Update 78 of Storyline 360 please contact our Support team here.
- 64-bit Storyline 360: We remain committed to shipping a 64-bit version of Storyline 360. We are currently formulating plans to validate the stability of the 64-bit version of Storyline 360. Once that is complete we’ll share details about our plans to roll this out for customers to use. I’ll share some of these details in next month’s update.
- Storyline 360 Private Beta: I’d like to close out this monthly update by extending an invitation to join our private beta program. The beta version of Storyline 360 is updated weekly with new features and bug fixes before the public version. Participants in this program can report bugs or provide feedback directly to the Storyline 360 engineering team. The beta version can also be installed side-by-side with the public one so you don't have to choose one or the other. If you’re interested in participating please e-mail beta@articulate.com and we’ll get you added to the program!
- The percentage of users on the latest update in its first 4 weeks post-release is consistent:
Welcome to the June 2023 “State of Storyline” update. This month I’d like to touch on our application error rate, customer reported issues, and pre-release quality assurance as well as share our progress on creating a 64-bit version of Storyline 360.
Application Error Rate
In Update 76 of Storyline 360 we addressed 12 of the errors most commonly encountered by users. These errors affected areas such as rendering audio waveforms on the timeline, encoding video, and writing files to disk at publish time.
Update 76 was released at the end of May and we’re currently monitoring our error reports as Storyline 360 users adopt the update. It typically takes 4 weeks for a new Storyline 360 version to be adopted widely enough to gather useful insights from this data, so we’ll share the impact these fixes had on our metrics in next month’s update.
Customer Reported Issues
Over the last month we have continued prioritizing bugs affecting 5 or more customer accounts. When last month’s update was published we were tracking 39 such bugs. Update 76 of Storyline 360 addressed 17 of these affecting areas such as text entry word wrapping behavior, hyperlinks in animated text, and importing tables from PowerPoint presentations.
We also addressed several other bugs that did not meet the criteria of impacting 5 or more customer accounts but were clearly still causing customer pain. These bugs affected areas like the new Background Audio feature, the Storyline 360 window failing to restore after being minimized, and the timeline play/pause/stop buttons becoming disabled unpredictably.
As of this writing we are tracking 23 bugs affecting 5 or more customer accounts and remain committed to continuing this work until they are resolved.
Pre-release Quality Assurance
We’ve continued making changes to our test and release cycles in an effort to improve the quality of each Storyline 360 release. This month I want to highlight a couple of areas that we’ve been focusing on recently:
- Release Validation: New versions of Storyline 360 are released on a monthly basis, which means we typically work in 4 week release cycles. In the past, some portion of the Storyline 360 engineering team would spend the last week of each release cycle testing and validating the new version to ensure it’s ready for release. As we’ve taken a critical look at how we approach quality in Storyline it has quickly become clear that having only a portion of the team participate in this release validation effort is not adequate. Starting with Update 76 of Storyline 360 we’ve had the entire team work together on pre-release validation. Having the whole team participate will not only help us catch more issues prior to release, but will also have us working with the application from a customer perspective more frequently. We’ve found that using Storyline in a manner similar to our customers helps us uncover minor bugs and usability issues that don’t tend to get a lot of attention through the normal bug triage and prioritization process.
- Test Automation: We employ a suite of automated tests to continuously validate the stability of the Storyline 360 application. Some of these tests only exercise one specific component of Storyline in isolation, while others run the full application and simulate a user performing common operations. These tests have replaced some of the repetitive manual testing that traditionally had to be done by a person. We’ve recently begun an effort to expand this test suite to better cover the Storyline 360 HTML5 Player application. With such a wide variety of devices, operating systems, and web browsers in use today, ensuring that the HTML5 Player continues to work with all of them can be a time consuming process for us to do manually. We are working to leverage modern testing frameworks and platforms to automate as much of this work as possible so that we can focus on more valuable tasks like reviewing customer bug reports, investigating frequently occurring errors, and finding new ways to improve the product.
64-bit Storyline 360
Over the past month we’ve kicked off the initial engineering work on a 64-bit version of Storyline 360. We’ve completed some of the initial infrastructure work to allow us to build and test a 64-bit version of Storyline 360 internally. We’ve also begun addressing some of the 32-bit third party components that will need to be updated, replaced, or removed.
Storyline 360 uses a Chromium-based web browser component for functionality like in-app preview, the 360° image editor, and publishing to video. We’ve successfully created a 64-bit version of this component and are working to integrate it into the 64-bit Storyline build.
We have also identified several different strategies for addressing other third party components that enable functionality like screen recording, spell check, and PowerPoint import. We will begin evaluating and implementing these strategies in the coming weeks.
Looking Forward
To close out this month’s update I’d like to share a couple of additional items:
- Storyline Quality Survey: We recently sent out a survey to some of our most prolific Storyline 360 users to gather feedback on the application’s overall quality. As responses to the survey come in we’ve been reviewing them to find common themes and direct our future efforts to improve and expand the product. I want to express our appreciation for anyone who received the survey and took the time to respond. If you were not among the Storyline 360 users who received an invitation to complete the survey but are interested in participating you can find and complete the survey here: Storyline Quality Survey. We will be following up with respondents in the coming weeks to discuss their specific feedback.
- Storyline 360 Private Beta: I would once again like to extend an invitation to join our private beta program. The beta version of Storyline 360 is updated on a weekly basis and gets new features and bug fixes before the public version. Participants in this program can report bugs or provide feedback directly to the Storyline 360 engineering team. The beta version can also be installed side-by-side with the public one so you don't have to choose one or the other. If you’re interested in participating please e-mail beta@articulate.com and we’ll get you added to the program!
As promised in my comment from September 2023, we’ve turned these monthly “State of Storyline” updates into a quarterly article series on E-Learning Heroes. You can find the first update here: Engineering Journal: Storyline Quality Update Q1 2024. I also published a separate article specifically about 64-bit Storyline 360: Engineering Journal: Demystifying 64-bit Storyline. I hope that you find these insightful. I will no longer be updating this post, but stay tuned to E-Learning Heroes for future updates!
