Storyline under a Microscope: An update from the CTO

Apr 11, 2023

Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where things stand and where they need to be.

There were a lot of positive signs that we already knew. For example, nearly a million courses were created with Storyline last year. But to be honest, we found things we didn’t like and were dismayed our heroes had to call us out to look deeper.

As an example of what we found, our error reporting shows us that 3% of application sessions hit an error. Based on some quick math, those errors add up to 3,000+ hours last year of course creator time spent facing errors. That was one example that surprised us. The other was “the state of defects” including the age of bugs, the number of outstanding defects reported by 5 or more customers, and the number of “bug workarounds” the community and our eLearning Heroes team come up with everyday.

As an engineering team, we’ve been very focused on new features, on new releases, and we relied on our QA and support partners to work directly with you all to keep us paying attention to the right things and keep our quality on track. But by being separated from our customers, we were missing the full picture.

We are committed to do the work to earn back your trust. We recognize that in our focus to build new capabilities, we took our eye off the foundations that matter to your success every day.

So here are the top three things we learned and what we plan to do about it. 

Quality First: We must prioritize Storyline quality over everything else.

  • First, we commit to prioritize quality over features. We recognize more features don’t help you succeed when you’re encountering paper cuts on a regular basis. Quality comes first. We’ve still got exciting new features on the horizon but we are doubling down on quality first.
  • Second, we're changing our test and release cycle. The team operates on a monthly release cycle, with the final week focused on validation. We release a beta build to a dedicated group of customers on a weekly basis so they can test and report any issues to our team. Our Engineering, QA, and Support teams have previously operated in separate yet parallel tracks, but we’ve moved to a partnership where they can lean on each other to learn, test, and focus on customer needs. The next few releases will focus on quality and deepening these partnerships to create a better Storyline. 
  • Third, we will source validation scenarios from the community. We heard from a number of you who say you wait months before upgrading Storyline and do extensive validation in advance. We will source validation scenarios from the community to include directly in our testing.
  • Fourth, we will invest in Storyline’s foundations. We are doing a deep dive into 64-bit support for Storyline and will report back with a plan. We had taken the stance that adding more capabilities to Storyline was job #1. So we were focused on delivering Storyline updates, along with quality updates, every month instead of disappearing into a cave on a long infrastructure project where the value of the impact is hard to measure until it is done. We are committed to investing in Storyline’s foundations so expect to hear more soon.

Transparency: We need to be more transparent about how we are prioritizing our work.

  • First priority is to reduce the error rate to below 1%. You can’t build courses, if you keep running into errors. In release 74, we fixed the top 18% of those errors that generated support cases but there is a lot more work to be done so those errors don’t escape our tests. We’ll report back monthly to update how we are doing on this critical metric.
  • Second we prioritized 35 older issues impacting five or more customers. We reviewed nearly 300 issues open over the life of Storyline. 35 of them are impacting five or more customers that were not at the top of our list. We are prioritizing those now and shipped a number of fixes for those in Release 74. Previously we focused our calories almost exclusively on more recent issues. So we’ve dedicated more engineers to focus on quality and renewed our focus on fixing older issues. As we work through those issues, we will share details.
  • Third, we are cleaning up our bug database. As a bug ages, finding the issue and fixing it without impacting something else becomes increasingly complex. When we reviewed all open issues, we found issues over 3 years old that affected one customer and had a workaround. We aren’t going to fix those issues. All defects are a problem and we don’t mean to imply we don’t care about quality. We need to focus our team on what impacts the community the most, and report back to those customers transparently about what we won’t fix instead of pretending we might some day.

Communication: We need to communicate proactively instead of reactively.

  • First, we will commit to monthly updates to the community from our Engineering team. We are working on the format and content to make it valuable, but to start we will commit to providing a “State of Storyline” monthly so you have the opportunity to know and react to what we are focused on. Quality is our number one priority right now and we are looking for measurable improvements that make a difference to you.
  • Second, our engineering team and I commit to working directly with all of you. Our engineers and engineering leaders have kept a distance from you all by accident. We’ve been focused on building and relied on our partners inside Articulate to be the “voice of our customers” rather than listening directly. So this is a commitment for us to work better together directly. We appreciate the time, energy, and passion you all bring to Storyline and we commit to make that worth it.

Storyline 360 is our flagship product and is how you help your employees, your customers, and your companies to learn and grow everyday. We have a lot of exciting things planned this year but agree that quality must come first. If you’re interested in joining our beta program or sharing your teams’ Storyline validation scenarios, please email beta@articulate.com.

Our team will be updating this thread directly each month to keep you aware of our progress. To share feedback on our process, our plans, and our areas of focus please open a new thread or reach out to the support team so they can connect us directly.

We are committed to  more transparency and proactive communication directly with the engineering team because honestly we are only here because of you.

1 Reply
Jesse Taber

Last month we shared our plans to address quality issues in Storyline 360 and committed to monthly updates in the interest of open, honest, and transparent communication with our customers.

My name is Jesse Taber, and I’m the Engineering Manager for the Storyline 360 Team. I’d like to share our progress in 3 key areas related to improving Storyline 360’s overall quality and stability: application error rate, customer-reported issues, and pre-release quality assurance.

Application Error Rate

You’re working hard on your course, you make a seemingly innocuous change, and suddenly you’re confronted with the “Articulate Storyline Error Report” dialog or even worse, the Storyline application crashes completely and has to be restarted. Not only are these errors an impediment to your work, but they shake your confidence in the application. For these reasons, our primary focus over the past month has been investigating and resolving these errors.

Last month Kerry mentioned that ~3% of Storyline sessions were encountering an error at least once. During the last 3 releases of Storyline 360 ~30% of users were encountering the error dialog at least once per month. Our goal is to drive both of these metrics down under 1%.

In Update 75 of Storyline 360, we addressed 9 of the most common errors that users were experiencing during their Storyline sessions. These errors affected a wide range of Storyline 360 functionality, including manipulating shape states, interacting with the ribbon, and editing or publishing videos. Update 75 was released on April 18, and we are seeing the following improvements to our average error rate metrics:

% of application sessions encountering the error dialog at least once: 2.7% (~0.3% improvement)

% of users encountering the error dialog at least once per month: 22.8% (~7.2% improvement)

While these metrics are promising, we recognize that we have a lot more work to do. We are devoting engineering resources to address these errors until we reach our goal.

On a related note: The Articulate Storyline Error Report dialog allows you to share additional information about the circumstances that led to the error. This goes directly to the Storyline 360 engineering team to use when investigating errors. It does not create a Support case for follow-up, but you can always connect with our Support team here.

Customer Reported Issues

We have been working to clean up our bug database. Our Engineering and Customer Support teams have been working closely together to dive deep on customer-reported issues to be sure that we have the most accurate picture possible of the issues that are impacting the most customers. This effort has led to many issues being de-duplicated and consolidated so that we have a much clearer idea of where to focus our bug-fixing efforts. Prior to this work we had identified 35 issues impacting 5 or more customer accounts based on support case data, but that number rose to 44 as a result of our de-duplication and consolidation.

In Update 75 of Storyline 360 we addressed 5 of these issues affecting 5 or more customers based on support case data, bringing that count down to 39. These issues impacted areas such as video compression, sliders, and performance in the Storyline 360 HTML5 player application.

Additionally, we addressed 5 issues that were reported and discussed by users in public forums such as Twitter or our own E-Learning Heroes forum. These issues impacted areas such as the focus order of objects on a slide, the “print results” action for quizzes, and an annoying bug where the Storyline 360 HTML5 Player would show the Resources tab even if the author had unchecked that option.

Pre-release Quality Assurance

We shared that we were making changes to our test and release cycles with an eye for improving the techniques used to validate quality as well as promote better alignment between the Engineering and Support teams. This effort will be continuously refined, but we have taken the following measures to start:

  • Establishing a formal Support Liaison role: The Support Liaison role was created to help bridge the gap between Engineering and Support and promote better cooperation between these teams. The Support Liaison works very closely with the Storyline 360 engineering team to ensure that we are hearing the voice of the customer. This role will share customer feedback with us (good, bad, and ugly), bring issues to our attention that might not yet have surfaced through our normal support case data analysis, and bring the support team's perspective on what issues the team should focus on next. The entire engineering team is focused on getting closer to our customers, and the Support Liaison is a big step in that direction.
  • Sourcing validation scenarios: Storyline 360 is a large, complex application with many features. We know that our customers use it to create interactive, entertaining, informative, and beautiful courses every day, but when we’re heads-down buried in the minutiae of the code on a daily basis, it can be easy to lose sight of the impressive ways our customers use the application every day. When we’re validating a new Storyline 360 release we try to use the product similarly to how our customers do. We’ve tapped into our very talented training team and asked them to define the common usage scenarios in Storyline. We now have a suite of common Storyline 360 course scenarios, such as conditional navigation, tabs interactions, and software simulations, that we use to put the application through its paces prior to releasing it publicly. We’ll continue to look for new validation scenarios from both the customer community and our own Storyline 360 experts to ensure we’re testing things in the right way.

Looking Forward

To wrap up this month’s update, I’d like to share things we’re actively working on:

  • 64-bit Storyline 360: We’re doing a deep dive on what it will take to create a 64-bit version of Storyline 360. Storyline is a large and complex piece of software comprised of many different components, both created internally by Articulate and sourced from third parties. Some of these components were not designed to work outside of a 32-bit process architecture and will need to be updated, replaced, or removed. We have identified the components that represent the largest amounts of effort and highest risk to the success of a 64-bit conversion. Our next step is to evaluate our options on a per-component basis and determine the plan for each. We will begin that effort this month and will share updates on our progress.
  • Storyline 360 Private Beta: Storyline 360 has a private beta program in which the software is updated weekly. The beta version gets new features and bug fixes before the public version does and serves as a great way for us to gather customer feedback or identify issues before the changes are made widely available. Any bug reports or feedback made through the private beta go directly to the Storyline 360 engineering team for review. Last month we extended an invitation for anyone interested in participating to e-mail beta@articulate.com, and we would like to remind everyone that this invitation still stands. Participating in the private beta helps make the product better and gives you an inside track on what changes are coming in future public releases.

This discussion is closed. You can start a new discussion or contact Articulate Support.