How End-to-End Testing moves us closer to Continuous Accessibility

In a recent article, we wrote in some depth about the concept of Continuous Accessibility. Here, we’ll talk about how end-to-end (E2E) testing can bring you closer to that ideal.

To recap, Continuous Accessibility (CA) is about helping companies make their web-based products more accessible to users of assistive technologies by facilitating routine, early detection of accessibility defects.

The ability to detect accessibility problems in the design phase of the software development life cycle (SDLC) is improving with the emergence of tools such as Stark for Figma, Adobe XD, etc. We love that because by catching potential problems as early as possible, fixes are relatively cheap to implement.

Once software is built, change becomes more expensive. 💸

One of the principles of CA is to “let computers perform repetitive tasks so people can solve hard problems.” Fortunately, many software teams are already accustomed to building automated testing into the software development lifecycle (SDLC) via Continuous Integration (CI) / Continuous Deployment (CD) pipelines such as GitHub Actions and GitLab CI/CD.

Static Code Analysis

The most commonly used tools used to find accessibility problems in CI/CD today are what are called static code analysis (SCA) tools¹, the best-known example of which is axe-core.

The major limitation of these SCA tools is that they can only identify around 25% of accessibility bugs². That’s not a knock on the tools, to be clear. That 25% number is hard-coded (so to speak) into the approach of analyzing source code rather than exercising the resulting user experience (UX).

And to be clear, we love axe-core and similar tools! You should totally integrate SCA into your CI/CD. SCA tools are cheap and fast and should absolutely be included in your CA strategy. Building the muscle of catching accessibility problems before they affect end users is key to CA. SCA has limitations to be sure, but it’s way better than doing nothing!

At the same time, we believe the SCA approach is not sufficient for achieving true CA, even with AI’s promises. End-to-end (E2E) testing has the potential to greatly increase the range of what automated tooling can do in pursuit of Melanie Sumner’s call for the “novel application of existing software engineering concepts” to the problem of CA.

End-to-End Testing

E2E testing has been around in the software industry for a long time. Like any tool, it has strengths and weaknesses. Sometimes called “integration” tests, they exercise user flows, not just independent components of a software system.

End-to-end testing, also known as E2E testing, is a way to make sure that applications behave as expected…. This type of testing approach starts from the end user’s perspective and simulates a real-world scenario.

CircleCI.com

An example of an E2E test might be one that simulates a user signing up for a website for the first time. Steps in the test could include the following:

1. Navigate to https://example.com/sign-up
2. Locate the username field, enter a value
3. Locate the password field, enter a value
4. Activate the “Sign Up” button
5. Expect a confirmation message

In contrast with SCA tools which are designed to work anywhere, E2E tests are hand coded for a specific website or app. This gives them crucial context to find bugs SCA misses. The tradeoff is that E2E tests are expensive to create (a human writes them). It would be unrealistic to write an E2E test for every page on a website.

Anyone who’s done any amount of manual auditing of websites for accessibility will tell you that the 5 seemingly innocuous steps in our preceding example are fraught with possible WCAG violations and UX problems. 😅

What does a screen reader announce when the page loads? How easy is it for the user to skip past complex top navigation and find the actual sign up form? Are the form fields clearly labeled? Are any error states announced? Can the “Sign Up” button be activated from a keyboard or only by mouse click? Where does focus land when the confirmation page loads?

SCA tools struggle with this kind of thing because they’re not able to factor in design intent. They can tell you whether the username field has an associated label, but they can’t tell you whether the label makes sense in the context of the page.

E2E tools have struggled with the complexity accessibility introduces too, but they don’t have to.

Accessibility E2E testing

As we mentioned earlier, E2E testing has been around for a long time and is commonly used in software development to ensure that user flows behave as expected. However, accessibility isn’t usually considered—partly because existing E2E tools lack some accessibility-specific capabilities and partly because “the end user’s perspective” has always been assumed to be that of a non-disabled user.

Traditional E2E tests written without accessibility in mind implicitly rely on emulated mouse or touch gestures and selectors based on elements’ visual appearance, as opposed to the programmatic appearance assistive technologies rely on (shoutout to testing-library for recognizing this problem 📣). A traditional E2E test provides high confidence that a mouse user can complete the flow being tested, but low-to-no confidence for keyboard or screen reader users.

An accessibility E2E test should start from the perspective of disabled users. It should include assistive technologies, accessibility-specific features, and usability guidelines such as WCAG.

When an accessibility E2E test passes, you should have high confidence that the user flow being tested can be completed using a keyboard, a screen reader like NVDA, or voice control. You should be able to trust that skip links, zoom, and accessible alternatives for interactions like drag-and-drop work. You should even get a picture of WCAG coverage. And this should happen on every code change, providing near-real time feedback for developers.

Accessibility E2E + SCA

For Continuous Accessibility, E2E and SCA should be used together to maximize coverage in CI/CD. E2E can find the keyboard traps and screen reader bugs that SCA can’t, but SCA can quickly scan hundreds or even thousands of pages.

The trick is to prioritize. Create accessibility E2E tests for the most critical areas of your website or app and employ SCA tools for the rest. Together, they can help you achieve a Continuous Accessibility strategy with the broadest coverage.

Our approach

As longtime believers in the power of software automation to scale impact, we set out to build a way to achieve high-confidence validation that user experiences are accessible at the speed and scale of automated testing. Learn more about our End-to-end Accessibility Testing service and reach out to schedule a demo.

We’re constantly learning, so stay tuned for more on the topic of Continuous Accessibility in future articles!

---

Footnotes:

1. There’s some debate over how to categorize accessibility checkers like axe-core. A linter like eslint-plugin-jsx-a11y is the classic SCA example—it only considers source code. “Checkers” like axe-core, WAVE, etc., don’t look at a website’s source code directly. Instead they look at the DOM after a browser renders the source. That said, they are static in the sense that the DOM is not manipulated from its initial state, at least, not automatically.
2. Source: Automated Accessibility Testing Tools: How Much Do Scans Catch? There’s plenty of debate over this number, many claim up to 50%. It’s tricky to pin down a specific number because the definition of “accessibility bugs” is up for interpretation. It’s usually an experience-based estimate of the % of bugs usually found by an automated scan when followed up by a manual audit. It doesn’t account for more complex sites having more potential bugs, nor the inverse.