Storyline under a Microscope: An update from the CTO
Apr 11, 2023
Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where things stand and where they need to be.
There were a lot of positive signs that we already knew. For example, nearly a million courses were created with Storyline last year. But to be honest, we found things we didn’t like and were dismayed our heroes had to call us out to look deeper.
As an example of what we found, our error reporting shows us that 3% of application sessions hit an error. Based on some quick math, those errors add up to 3,000+ hours last year of course creator time spent facing errors. That was one example that surprised us. The other was “the state of defects” including the age of bugs, the number of outstanding defects reported by 5 or more customers, and the number of “bug workarounds” the community and our eLearning Heroes team come up with everyday.
As an engineering team, we’ve been very focused on new features, on new releases, and we relied on our QA and support partners to work directly with you all to keep us paying attention to the right things and keep our quality on track. But by being separated from our customers, we were missing the full picture.
We are committed to do the work to earn back your trust. We recognize that in our focus to build new capabilities, we took our eye off the foundations that matter to your success every day.
So here are the top three things we learned and what we plan to do about it.
Quality First: We must prioritize Storyline quality over everything else.
- First, we commit to prioritize quality over features. We recognize more features don’t help you succeed when you’re encountering paper cuts on a regular basis. Quality comes first. We’ve still got exciting new features on the horizon but we are doubling down on quality first.
- Second, we're changing our test and release cycle. The team operates on a monthly release cycle, with the final week focused on validation. We release a beta build to a dedicated group of customers on a weekly basis so they can test and report any issues to our team. Our Engineering, QA, and Support teams have previously operated in separate yet parallel tracks, but we’ve moved to a partnership where they can lean on each other to learn, test, and focus on customer needs. The next few releases will focus on quality and deepening these partnerships to create a better Storyline.
- Third, we will source validation scenarios from the community. We heard from a number of you who say you wait months before upgrading Storyline and do extensive validation in advance. We will source validation scenarios from the community to include directly in our testing.
- Fourth, we will invest in Storyline’s foundations. We are doing a deep dive into 64-bit support for Storyline and will report back with a plan. We had taken the stance that adding more capabilities to Storyline was job #1. So we were focused on delivering Storyline updates, along with quality updates, every month instead of disappearing into a cave on a long infrastructure project where the value of the impact is hard to measure until it is done. We are committed to investing in Storyline’s foundations so expect to hear more soon.
Transparency: We need to be more transparent about how we are prioritizing our work.
- First priority is to reduce the error rate to below 1%. You can’t build courses, if you keep running into errors. In release 74, we fixed the top 18% of those errors that generated support cases but there is a lot more work to be done so those errors don’t escape our tests. We’ll report back monthly to update how we are doing on this critical metric.
- Second we prioritized 35 older issues impacting five or more customers. We reviewed nearly 300 issues open over the life of Storyline. 35 of them are impacting five or more customers that were not at the top of our list. We are prioritizing those now and shipped a number of fixes for those in Release 74. Previously we focused our calories almost exclusively on more recent issues. So we’ve dedicated more engineers to focus on quality and renewed our focus on fixing older issues. As we work through those issues, we will share details.
- Third, we are cleaning up our bug database. As a bug ages, finding the issue and fixing it without impacting something else becomes increasingly complex. When we reviewed all open issues, we found issues over 3 years old that affected one customer and had a workaround. We aren’t going to fix those issues. All defects are a problem and we don’t mean to imply we don’t care about quality. We need to focus our team on what impacts the community the most, and report back to those customers transparently about what we won’t fix instead of pretending we might some day.
Communication: We need to communicate proactively instead of reactively.
- First, we will commit to monthly updates to the community from our Engineering team. We are working on the format and content to make it valuable, but to start we will commit to providing a “State of Storyline” monthly so you have the opportunity to know and react to what we are focused on. Quality is our number one priority right now and we are looking for measurable improvements that make a difference to you.
- Second, our engineering team and I commit to working directly with all of you. Our engineers and engineering leaders have kept a distance from you all by accident. We’ve been focused on building and relied on our partners inside Articulate to be the “voice of our customers” rather than listening directly. So this is a commitment for us to work better together directly. We appreciate the time, energy, and passion you all bring to Storyline and we commit to make that worth it.
Storyline 360 is our flagship product and is how you help your employees, your customers, and your companies to learn and grow everyday. We have a lot of exciting things planned this year but agree that quality must come first. If you’re interested in joining our beta program or sharing your teams’ Storyline validation scenarios, please email firstname.lastname@example.org.
Our team will be updating this thread directly each month to keep you aware of our progress. To share feedback on our process, our plans, and our areas of focus please open a new thread or reach out to the support team so they can connect us directly.
We are committed to more transparency and proactive communication directly with the engineering team because honestly we are only here because of you.
This discussion is closed. You can start a new discussion or contact Articulate Support.