Forum Discussion
Storyline under a Microscope: An update from the CTO
Over the past three weeks, the engineering team put Storyline under a microscope. We dug into bugs, dug into our data, and heard a ton of direct feedback to sharpen our interpretation of where thing...
A Year of Change: Reflecting on 12 months of Storyline development
Since stepping into the CTO role at Articulate in early 2023, I've had the privilege of diving deep into our customer experiences. Back then, it quickly became clear that while there was a lot of love for Storyline, there were valid frustrations around unresolved bugs, lack of transparency, missing 64-bit support, and insufficient communication from our engineering team.
Hearing you loud and clear
About a year ago, your feedback was unmistakable: we needed to listen better. I shared our engineering teams’ commitment to change in a post titled, “Storyline under a Microscope: An update from the CTO”, and I'm thrilled to talk about the strides we've made since.
Our Shift in Focus
Today, our conversations with customers are future-focused, revolving around quality, advanced capabilities, and accessibility. It’s a shift from frustration to anticipation, and while we haven’t solved every issue, the progress is undeniable. So let’s review the key themes from the past year and highlight the progress we made on the journey.
Reflecting on Key Themes: Quality, Transparency, and Communication
Quality became focus number one for our team. In a nutshell, we hit pause on all new features to zero in on fixing existing issues, aligning our teams more closely for better quality control, diving into real customer scenarios to guide our releases, and doubling down on modernizing Storyline, including launching a 64-bit version in beta. In more detail:
- Prioritizing Quality Over Features: We took a strategic pause on developing new features for Storyline for six months, dedicating this period solely to enhancing product quality. The entire Storyline Engineering Team shifted their focus to addressing and resolving existing quality issues, ensuring a stronger foundation for our product.
- Unifying our Approach for Better Results: Previously, our Engineering, Quality Assurance, and Support teams worked independently on Storyline releases, which sometimes led to siloed efforts. We've since revamped our approach, bringing these teams into a cohesive unit. This collaboration ensures we can identify and address quality issues more efficiently, resulting in faster and more effective solutions.
- Learning Directly From User Experiences: In partnership with David Anderson, our Director of Customer Training, we've integrated real-world course-building scenarios into our testing process. These scenarios, reflective of our customers' daily challenges, are now a crucial part of validating each Storyline update before it goes live. This practice has not only improved our issue detection but has also deepened our understanding of what our users truly need.
- Modernizing Storyline: A year ago, we reinforced our commitment to Storyline by launching the first 64-bit version, Storyline 360 x64, in a public beta. We're now in the final stages of preparing Storyline 360 x64 for widespread release and are actively working on additional modernization projects to ensure Storyline continues to meet the evolving needs of our users.
Along with our investment in quality, we recognized the need to increase transparency to clarify how we make decisions around what we work on.
Commitment to Transparency
Starting last year we committed to being more transparent about how we approach building and maintaining Storyline. We opened up about our decision-making process, our progress toward reducing unexpected errors, and our plans for addressing long-standing bugs. Our revamped bug triage process ensures we're responsive and focused on what matters most.
- Reducing the error rate: When we started this effort in April of 2023, about 3% of Storyline sessions were encountering an unexpected error. We committed to getting this metric under 1%. As of March 1, 2024 1.2% of Storyline sessions are encountering an unexpected error. We’ve worked hard to reduce these errors and we won’t stop until we get under our 1% goal because any error that interrupts your day is like a “paper cut” in your way.
- Staying “below five”: Twelve months ago, we discovered we had stopped paying attention to “older bugs”. So we focused our attention on not just recent issues but all bugs that had five customers or more. We have 100s of thousands of users, and set a line in the sand that all bugs must stay “below five customers” otherwise we stop feature development to fix. We cleaned up our backlog of bugs affecting five or more customers and Storyline is much stronger for it.
- Revamping our bug triage process: Our bug database was, admittedly, a bit of a mess in April of 2023 and it was difficult to find signal in the noise. The right things were not always getting prioritized. So along with jumping on those older bugs, we revamped our daily bug triage approach to prioritize new bugs in the latest releases. We also started tracking the overall defect rate coming in from support cases to drive it below 10%. These changes helped us jump on issues causing pain quickly so that it didn’t spread.
To reinforce our quality and transparency commitment, we recognized the need to increase communications to ensure your concerns were being heard and addressed.
Emphasis on continuous communication
In summary, monthly updates last year have kept you informed. We’ve increased our engagement with you through direct conversations and expanded our beta program, ensuring your voice is heard and valued.
- Monthly updates from Engineering leadership: Storyline’s engineering leader, Jesse Taber, provided monthly “State of Storyline” updates as comments on my original E-Learning Heroes post. In Jesse’s final update in September 2023 after we took Storyline out of Code Red, he committed to continue providing updates on a quarterly basis.
-
- The first quarterly Storyline quality update is now available.
As a bonus, Jesse also wrote a separate article all about 64-bit Storyline.
- The first quarterly Storyline quality update is now available.
- Commitment to working directly with you: We spoke with a lot of customers last year to learn about their experiences using Storyline 360 and have made that a regular practice. The insights from those conversations are invaluable. We also expanded participation in our private beta program by 55%! We love working directly with our private beta customers to gather feedback about new features and hear about issues. If you’re interested in a more direct line to our engineering team, please e-mail beta@articulate.com.
Looking Forward
As we move forward, balancing innovation with quality remains our unwavering commitment. Vanessa Fage, leading our Storyline quality team and beta program, is a testament to our dedication to continuous improvement and open communication.
A Year of Collaboration
This journey has reinforced the value of working directly with you, a lesson we'll carry forward to ensure we never stop listening again. Your trust fuels our progress, and for that, we’re profoundly grateful.
Thank you for inspiring us and for being a pivotal part of Storyline's evolution.
Last month we shared our plans to address quality issues in Storyline 360 and committed to monthly updates in the interest of open, honest, and transparent communication with our customers.
My name is Jesse Taber, and I’m the Engineering Manager for the Storyline 360 Team. I’d like to share our progress in 3 key areas related to improving Storyline 360’s overall quality and stability: application error rate, customer-reported issues, and pre-release quality assurance.
Application Error Rate
You’re working hard on your course, you make a seemingly innocuous change, and suddenly you’re confronted with the “Articulate Storyline Error Report” dialog or even worse, the Storyline application crashes completely and has to be restarted. Not only are these errors an impediment to your work, but they shake your confidence in the application. For these reasons, our primary focus over the past month has been investigating and resolving these errors.
Last month Kerry mentioned that ~3% of Storyline sessions were encountering an error at least once. During the last 3 releases of Storyline 360 ~30% of users were encountering the error dialog at least once per month. Our goal is to drive both of these metrics down under 1%.
In Update 75 of Storyline 360, we addressed 9 of the most common errors that users were experiencing during their Storyline sessions. These errors affected a wide range of Storyline 360 functionality, including manipulating shape states, interacting with the ribbon, and editing or publishing videos. Update 75 was released on April 18, and we are seeing the following improvements to our average error rate metrics:
% of application sessions encountering the error dialog at least once: 2.7% (~0.3% improvement)
% of users encountering the error dialog at least once per month: 22.8% (~7.2% improvement)
While these metrics are promising, we recognize that we have a lot more work to do. We are devoting engineering resources to address these errors until we reach our goal.
On a related note: The Articulate Storyline Error Report dialog allows you to share additional information about the circumstances that led to the error. This goes directly to the Storyline 360 engineering team to use when investigating errors. It does not create a Support case for follow-up, but you can always connect with our Support team here.
Customer Reported Issues
We have been working to clean up our bug database. Our Engineering and Customer Support teams have been working closely together to dive deep on customer-reported issues to be sure that we have the most accurate picture possible of the issues that are impacting the most customers. This effort has led to many issues being de-duplicated and consolidated so that we have a much clearer idea of where to focus our bug-fixing efforts. Prior to this work we had identified 35 issues impacting 5 or more customer accounts based on support case data, but that number rose to 44 as a result of our de-duplication and consolidation.
In Update 75 of Storyline 360 we addressed 5 of these issues affecting 5 or more customers based on support case data, bringing that count down to 39. These issues impacted areas such as video compression, sliders, and performance in the Storyline 360 HTML5 player application.
Additionally, we addressed 5 issues that were reported and discussed by users in public forums such as Twitter or our own E-Learning Heroes forum. These issues impacted areas such as the focus order of objects on a slide, the “print results” action for quizzes, and an annoying bug where the Storyline 360 HTML5 Player would show the Resources tab even if the author had unchecked that option.
Pre-release Quality Assurance
We shared that we were making changes to our test and release cycles with an eye for improving the techniques used to validate quality as well as promote better alignment between the Engineering and Support teams. This effort will be continuously refined, but we have taken the following measures to start:
- Establishing a formal Support Liaison role: The Support Liaison role was created to help bridge the gap between Engineering and Support and promote better cooperation between these teams. The Support Liaison works very closely with the Storyline 360 engineering team to ensure that we are hearing the voice of the customer. This role will share customer feedback with us (good, bad, and ugly), bring issues to our attention that might not yet have surfaced through our normal support case data analysis, and bring the support team's perspective on what issues the team should focus on next. The entire engineering team is focused on getting closer to our customers, and the Support Liaison is a big step in that direction.
- Sourcing validation scenarios: Storyline 360 is a large, complex application with many features. We know that our customers use it to create interactive, entertaining, informative, and beautiful courses every day, but when we’re heads-down buried in the minutiae of the code on a daily basis, it can be easy to lose sight of the impressive ways our customers use the application every day. When we’re validating a new Storyline 360 release we try to use the product similarly to how our customers do. We’ve tapped into our very talented training team and asked them to define the common usage scenarios in Storyline. We now have a suite of common Storyline 360 course scenarios, such as conditional navigation, tabs interactions, and software simulations, that we use to put the application through its paces prior to releasing it publicly. We’ll continue to look for new validation scenarios from both the customer community and our own Storyline 360 experts to ensure we’re testing things in the right way.
Looking Forward
To wrap up this month’s update, I’d like to share things we’re actively working on:
- 64-bit Storyline 360: We’re doing a deep dive on what it will take to create a 64-bit version of Storyline 360. Storyline is a large and complex piece of software comprised of many different components, both created internally by Articulate and sourced from third parties. Some of these components were not designed to work outside of a 32-bit process architecture and will need to be updated, replaced, or removed. We have identified the components that represent the largest amounts of effort and highest risk to the success of a 64-bit conversion. Our next step is to evaluate our options on a per-component basis and determine the plan for each. We will begin that effort this month and will share updates on our progress.
- Storyline 360 Private Beta: Storyline 360 has a private beta program in which the software is updated weekly. The beta version gets new features and bug fixes before the public version does and serves as a great way for us to gather customer feedback or identify issues before the changes are made widely available. Any bug reports or feedback made through the private beta go directly to the Storyline 360 engineering team for review. Last month we extended an invitation for anyone interested in participating to e-mail beta@articulate.com, and we would like to remind everyone that this invitation still stands. Participating in the private beta helps make the product better and gives you an inside track on what changes are coming in future public releases.
