Engineering Journal: Storyline Quality Update Q3 2024
Published on October 25, 2024 Hello E-Learning Heroes community members! I’m Jason Famularo, a Senior Software Engineer on the Storyline 360 team. One of my responsibilities is improving the quality of the Storyline 360 application, and that’s perfect because I’m here to give the Q3 Storyline quality update. This month, I’d like to spotlight the team’s quality process, which we use to validate new versions of Storyline 360 before they are released. As always, we’ll also review our quarterly Quality Metrics. Spotlight on Storyline 360 Quality Procedures Before we dive in, let’s ask, “What exactly is quality?” To quote the IEEE, “Software quality is a complex topic,” but in short, it describes the “reliability, usability, performance, and security” of a software application. All software teams want their products to function correctly, be easy to use, perform well, and certainly be secure. If a team doesn’t have a quality process defined, low-quality software will result. When I started here at Articulate, my manager, Jesse Taber, told me that “everyone owns quality.” This is a crucial mantra, because the focus on quality happens during each step of the process as a feature is designed, implemented, changed, tested, and maintained. So how do we ensure high-quality software on the Storyline 360 team? Storyline 360 Quality Process Overview The team’s quality process includes the following steps: Automated testing during development Acceptance testing when a feature is complete or updated or a bug is fixed (aka QA) Daily internal build testing and validation Weekly private beta releases Final release validation Post-release monitoring Let’s look at each step in more detail! Automated Testing Storyline 360 utilizes two different automated testing strategies: unit testing and integration testing. Unit testing is additional code that tests small, individual components of the application. This very powerful tool alerts software engineers when a small, seemingly unrelated change introduces a bug. Articulate has over 15,000 unit tests! While that sounds like a lot, each one runs in a fraction of a second, so the total time needed is just ten to fifteen minutes. Unit tests cover a wide gamut of functionality in Storyline 360, such as creating, opening, and saving project files, adding and editing content in the project, publishing courses, and creating course output. Integration testing is a layer above unit testing. This style of testing is focused on testing modules and components with each other. This ensures that the application as a whole works well with itself. A change in one module might have a downstream impact on others. For example, the new AI Assistant features in Storyline introduced an AI Assistant side panel. Our integration tests can catch an issue where the new AI Assistant side panel might introduce a bug with the Triggers or Comments panel, which we might have otherwise overlooked. Before a software engineer’s changes can be permanently added into the codebase, all unit tests must pass. Depending on the complexity of the change, we may also run integration tests at this time. If we don’t run them here, they always run at least once daily due to how long they take to complete. Acceptance Testing Along with our automated tests, we use acceptance testing. Before a software engineer delivers their work, they write specific steps that cover how their work should be tested, including both obvious and not-so-obvious ways the code may break. This will also cover the steps they took to test the code and any other areas that should be retested as a result. The software engineer then assigns their changes to another engineer for code review and testing. QA engineers monitor each change as well and assess the need for additional testing. For higher-risk changes, a QA engineer is assigned to do additional testing to mitigate that risk. The assigned engineers execute the test steps written by the authoring engineer and also perform additional testing they deem necessary. Edge cases are exercised at this time, and the tester attempts to use the feature in every way possible: with just a mouse and with just a keyboard, in various languages, undoing and redoing, using alternate avenues of accessing features—such as right-clicking—and just generally trying to find scenarios the author may have missed in their testing. The tester documents how they exercised the functionality. If issues are found, the authoring engineer fixes them, and the process repeats. Once the changes are tested satisfactorily, they are merged into the codebase. Once this is done, integration tests are run automatically. If anything fails, the engineer determines why and provides fixes immediately. A majority of bugs and issues are found during this stage. Internal Testing Once a day, usually overnight, a new version is created and given to Articulate employees for early access and testing. Other teams use these internal builds to access unreleased features and verify bug fixes. Other companies may call this process alpha testing or dogfooding, and while we love creatures with paws here at Articulate, we just call it internal testing. Any crashes that happen in this version are logged and sent to the Storyline 360 team, where a software and QA engineer review and determine the course of action to take. We’ll usually reach out to our coworkers who experienced the crash and gather more information. Private Beta In addition to internal testing, we have a private beta for customers like you! We release new builds weekly after testing has been completed and we are confident that the beta version is stable. Private beta releases don't get as much internal scrutiny as major releases, so bugs or other issues may surface—if you do find an issue, you’ll have a direct line to the engineering team (or, more accurately, we will have a direct line to you and will be in touch if you run into issues or crashes). We actively monitor feedback and telemetry data from the private beta to ensure the upcoming release behaves as expected. If you are interested in participating in the private beta, please email us atbeta@articulate.com. We receive great feedback from our users who are in the beta and are truly thankful for their efforts in helping us make a better product for everyone! Final Release Candidate Validation Storyline 360 has an ideal release cadence of once a month (sometimes it’s five or six weeks). The final week of the release cycle is called Overview Week and consists of a team-wide effort to find and stamp out any remaining issues. We create a release candidate build 7 to 10 days before we release it to users and use that for the Overview Week testing. The first step of Overview Week is reviewing all the code changes included in the next release. Three senior engineers tackle this task and identify risky changes that require additional testing or focus. Once the week begins, all changes made to the release are subject to a full regression test. The development team divvies up new features and bug fixes, and they retest each change again to find anything missed the first time and to ensure that it works with all the other changes in the release. While that happens, the QA team does end-to-end testing of all new features along with testing the output of published courses to ensure the new changes don’t break existing functionality. We maintain a list of key features and tricky scenarios and retest them in various web browsers. As the week goes on, the team creates courses that mimic features our customers use to create their amazing courses. For example, tabs interactions are a popular way for course authors to create dynamic and compelling content, and we make a course featuring tabs to ensure things are still working. Background audio, a way to give your course a soundtrack, is another example of a popular feature we exercise during our Overview Week. If an issue is found at any point, software and QA engineers review, resolve, and retest them. The final release usually happens on a Tuesday morning (in the United States). Post-Release Monitoring About 15% of Storyline 360 active users install a new release within the first week of its availability. By the end of the second week, about 30% of active users have installed the update. If there are any major issues in a release, we typically find them in the first two weeks, before the majority of users have adopted the release. The support and QA teams monitor incoming bug reports and work quickly to determine the cause and plot a course of action. The QA engineers also monitor the feedback users leave when they downgrade to a previous version of Storyline 360. This feedback often contains insight about new issues and helps us expedite fixes. Software engineers monitor telemetry data, and the support team monitors support cases and forums. If anything looks problematic, we address it as soon as possible. If the issue is particularly gnarly, we do a service release, which is a new release with critical bug fixes. Summary The Storyline 360 team follows a multistep process for each release to ensure product quality. We know it’s frustrating when issues creep into the release, so we are constantly working to improve our methodologies and to catch things as soon as possible. Quality Metrics Let’s review our quarterly quality metrics! Application Error Rate The application error rate measures how often Storyline 360 displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions—or unique instances of opening and then later closing Storyline 360—and users. Our goal is to get this metric under 1% for both. The application error rate for Storyline sessions has been hovering around 1.15% for the past six months and has continued to hold steady. Storyline 360’s application error rate per session for updates 84 through 90. The graph below tracks the percentage of users who encounter an error in a release. When Jesse first reported the application error rate by user for update 84 last year it was around10%, but has since risen to closer to 20%. While the session application error rate tends to stabilize the longer a given release is available, the user level rate often climbs. That’s because as more people adopt a new update, the chance that they encounter at least one unexpected error gets higher. We’re still working hard to address unexpected errors in Storyline to improve this metric—it’s been a focus for the past six months—and we are starting to see the results of our work. Downgrades This metric tracks how often a Storyline 360 user updates to a new version of the application—only to downgrade later to an earlier version. We interpret downgrades as an indication that authors encountered issues in a new version that prevented them from completing their work. Last year, we saw this metric dip below 1% at the end of the second quarter and remain there through the middle of the third quarter. Since then, it has climbed and seesawed between 1% and 2%, with a recent spike back to 2%. This spike is due to an increase as we work to remove old third-party dependencies in our Modern Player (so we can keep it Modern 😉). We’ve addressed these issues with a series of service releases, and expect downgrades to return to lower levels in the coming months. Collecting feedback when users downgrade to a previous version of Storyline 360 was very helpful here. Defect Rate This metric tracks the percentage of open support cases associated with an unfixed bug. An increase in this number is a signal that our support team is spending time fielding bug reports instead of helping customers get the most out of our products, so our goal is to keep this value below 10%. This metric has been below the 10% threshold for quite some time now, but it’s getting close to that threshold again. This is also due to the Modern Player dependency removal discussed in the previous section. We’ve addressed many of these issues and will continue to do so if any new ones come in. We rely on support cases to direct our bug-fixing efforts, so I encourage you tocontact our support team if you’re experiencing issues with Storyline 360. Publishing Failures This metric tracks the number of users who get an error during publishing. If you enjoy reading our engineering journals and have a great memory, you may recall this number was around 4.25% last quarter. What happened?! In short, we realized that we were unfairly penalizing ourselves by counting attempts to publish that were canceled by the author. We’ve made a significant effort over the past six months to address most of the top publishing failures. While we haven’t seen a drop in these error percentages yet, we expect to see one because adoption of a new version of Storyline 360 often takes a few months. Release 92 introduced redesigned publishing success and failure dialogs. If publishing fails with a problem the user can address, the user is told what happened and steps they can take to resolve it, an example is the running out of disk space message. Additionally, if publishing fails on a particular slide, we call out that slide so you can look at it and figure out what went wrong, be it a recent change or complex feature. We want you to get unblocked as soon as possible! We’ve got some work left to do to reach our goal of less than 1%. Incomplete Sessions This metric tracks how often Storyline 360 quits unexpectedly due to an error. Our goal is to maintain this metric under 1%. The Storyline team spent Q1 focused on improving this metric as much as possible. The percentage peaked at 3.8% at the end of Q4 last year, and those efforts have slowly made a big difference, as the percentage has steadily dropped this year. We will continue to monitor this closely and revisit it in the future as needed as we aim to achieve our goal. Wrap-Up Due to the lag in the number of users upgrading Storyline each month, we often see that our efforts take months to materialize in these metrics. It can be worrisome as we wonder, Did all that work do anything?! But as we see with reducing the number of incomplete sessions, that work does matter and eventually shows up in the quality statistics we track. We spent the past six months focusing on the publishing failure metric. As the bulk of that work is in the more recent updates, we should see improvements in those metrics soon. If we don’t, we’ll revisit publishing failures and resolve troublesome bugs quickly! In the meantime, if there are any topics you’d like to see covered in these quality updates, please reach out to the team atstoryline-engineering@articulate.com.Engineering Journal: Storyline Quality Update Q2 2024
Published on June 25, 2024 Hello E-Learning Heroes community members! It’s Jesse Taber, Director of Engineering, and I’m back with the Q2 2024 Storyline quality update. This quarter I want to start by spotlighting one area where we’re currently focused on improving quality—course publishing—before providing updates on the typical quality metrics. Let’s dig in! Spotlight on Publishing Quality We’ve been tracking how often publishing fails in Storyline 360 for over a year now. In the spring of 2023, we made impressive progress in reducing this metric, but we have since regressed. Line chart showing Storyline 360’s publishing failure percentage for updates 74 through 86. Before I explain why this happened and how we plan to continue reducing publishing failures, I’d like to walk you through what’s going on in the background when you publish your project—so you understand why the process sometimes fails. How Does Course Publishing Work? When you publish a course, Storyline reviews every scene and slide and builds data files describing everything that needs to happen to recreate the author’s vision for their course in a learner’s browser. Every object, trigger, animation, timeline cue point, image, video, etc., gets represented as a component in these data files. Additionally, images, video, and audio may need to be compressed or re-encoded in order to work properly in a variety of browsers. Next, the course player settings are distilled into separate data files that describe the player options, colors, text labels, etc. Finally, some specialized formats like publishing to Word or Video do extra work to create the final published Word or MP4 video file. As you can see, publishing a single Storyline course requires creating dozens of files and can be quite complex. And the more complex a process is, the higher the failure rate tends to be. How Do We Calculate the Publish Failure Rate? To calculate the publish failure rate, we divide the number of publish operations that failed by the total attempted. How do we know when publishing fails? It’s simple. When you publish your course, we record some basic information about it like the audio and video quality settings, the player features that are enabled, and the output type (Web, LMS, Video, etc.). As it progresses, we record other information related to performance and, most important, whether it finished successfully. Why Did the Publish Failure Rate Spike in July 2023? When fixing a publishing-related bug, we discovered that we weren’t capturing every failed publish. Some failed attempts to publish were being counted as intentional cancellations. When we fixed this issue and released it in Update 78, we knew we’d likely see a rise in publishing failures since we’d be capturing all of them correctly. While it was disheartening to see this metric increase, we knew it was important to have a complete and accurate picture of what was happening for authors using Storyline in their day-to-day work. Why Does Publishing Fail? But even with more accurate publishing failure metrics we could only tell how often publishing fails—not why it failed. To understand why, we had to correlate the failed publishing metrics with the error report data we get from users who submit this form: When we receive this data, we can tag it with keywords that allow us to recognize patterns, pinpoint issues, and fix them. When we started digging into publishing error reports, we discovered that several of the most common failures were related to disk input/output operations (disk I/O). What does that mean? Remember earlier when we talked about how complex the publishing process is? Creating so many files requires Storyline to write and read a lot of data to and from the computer’s hard drive, which is commonly referred to as disk I/O. Unfortunately, anytime software has to perform disk I/O operations, things can go wrong. For example: Your computer can run out of disk space. Storyline might be contending with another process to read or write a given file on disk. Background processes that scan or copy files, such as anti-malware or cloud backup services, often cause this kind of contention. An intermittent operating system issue could cause a disk I/O operation to fail unexpectedly, as was the case with a Windows update that rolled out in the spring of 2023. Disk I/O failures are often outside Articulate’s control and might require your help to resolve. For example: If you’re running out of disk space, you’ll need to free some up in order for Storyline to successfully publish your course. If Storyline encounters contention with another process when trying to read or write a file, sometimes waiting a split second and trying again works. Other times a particularly stubborn scanning service might prevent us from accessing a file for a long time. In these cases, you may need to stop the service temporarily or exclude certain files and folders from being scanned. But how do you, as the author, figure out why it’s failing? Unfortunately, right now there’s no easy way to do that. But that’s where this next section comes in. Next Steps We know how frustrating it is when Storyline fails to publish your course. And that frustration is only compounded by the fact that the error report dialog tells you something went wrong but doesn’t say why or how to fix it. That’s why we’re working to redesign this experience with an eye for providing you with more context about why the publish operation might have failed. This context will include information about whether the failure might be something the author could resolve themselves (e.g., free up disk space) as well as the specific scene and slide that were being published when the failure occurred. Of course, not all errors that happen during publishing will require the author to intervene. Many of these errors represent legitimate bugs in Storyline that we need to fix. We’re currently addressing the top 10 such errors to help ensure you can publish your projects successfully. Quality Metrics Application Error Rate The application error rate measures how often Storyline 360 displays the “Articulate Storyline Error Report” dialog. We track this data for both Storyline 360 sessions—or unique instances of opening and then later closing Storyline 360—and users. Our goal is to get this metric under 1% for both Storyline sessions and users. As we entered Q1 of 2024 the application error rate for Storyline sessions was hovering at around 1.15% and has continued to hold steady there. Line chart showing Storyline 360’s application error rate per session for updates 81 through 87. When I first reported the application error rate by user for Update 84 last quarter it was around 10%, but has since risen closer to 20%. While the session application error rate tends to stabilize the longer a given release is available, the user level rate often climbs. That’s because as more people adopt a new update, the chance that they encounter at least one unexpected error gets higher. We’re working hard to address unexpected errors in Storyline to improve this metric. Line chart showing Storyline 360’s application error rate per user for updates 81 through 87. Downgrades This metric tracks how often a Storyline 360 user updates to a new version of the application—only to downgrade later to an earlier version. We interpret downgrades as an indication that authors encountered issues in a new version that prevented them from completing their work. Last year, we saw this metric dip below 1% at the end of the second quarter and remain there through the middle of the third quarter. Since then, it has climbed and seesawed between 1% and 1.6%, with Update 84 dipping under 1%. While we’re happy to have reduced this number from the 2% we saw last year, we want to understand better why our customers downgrade. We’ll be working to clarify this in the coming months. Line chart showing Storyline 360’s downgrade percentage for updates 81 through 87. Defect Rate This metric tracks the percentage of open support cases associated with an unfixed bug. An increase in this number is a signal that our support team is spending time fielding bug reports instead of helping customers get the most out of our products, so our goal is to keep this value below 10%. This metric has been below the 10% threshold for quite some time now, but we will continue to monitor it closely and take action if it suddenly spikes due to an influx of reports about a bug. We rely on support cases to direct our bug-fixing efforts, so I encourage you to contact our support team if you’re experiencing issues with Storyline 360. Line chart showing Storyline 360’s defect rate for October 2023 through April 2024. Incomplete Sessions This metric tracks how often Storyline 360 quits unexpectedly due to an error. Our goal is to maintain this metric under 1%. The Storyline team spent Q1 focused on improving this metric as much as possible. It’s been hovering around 3.5% since Q4 of last year, and our efforts to improve it have been slow to progress. Every single instance that contributes to this metric requires a lot of time to investigate and fix, and each fix yields a marginal improvement to the overall rate. I reduced the range of values on the vertical axis of this chart to help illustrate the progress that was made last quarter. At the beginning of Q1 we saw a rate of ~3.8%. As we headed into Q2, that rate had dropped to ~3.3% for an improvement of 0.5%. We have had to direct some of our quality efforts into other areas recently, but we will continue to monitor this closely and revisit it in the future. Line chart showing Storyline 360’s incomplete session percentage for updates 81 through 87. Wrap-Up We’re currently focused on improving the publishing failure metric. I’ll provide an update on our progress in next quarter’s article. In the meantime, if there are any topics you’d like to see covered in these quality updates, please reach out: jtaber@articulate.com.
