Again, I'm Andrew, one of the co-founders of Assistive Labs. I'm going to turn it over to Weston and Nick.
Sounds good. Thanks all so much. For joining.
Pop in here with a few slides and then hopefully spend a bunch of time with examples and And demos.
So agenda for today, I'm going to do Just a brief extra And context.
Context about the accessibility space as we see it. And we'll talk about end-to-end testing that's not accessibility specific and what we mean by that.
Get into accessibility first, end-to-end testing. Followed by a demo and we'll try to leave plenty of time for questions and answers.
If you have questions along the way, hopefully you do. Please feel free to ask in the Zoom chat. Angel helped field those. We'll try to answer async and save some at the end to talk about.
So questions quite welcome. Bit about us, Andrew introduced uh himself. We also have Nick here.
Software engineers on our team And I'm Weston, the co-founder.
So a bit about Assistive Labs. We are an accessibility company.
Through and through. We really care about trying to raise accessibility quality Raise the bar on that through both tooling, but also processes. And we're big believers that human judgment is a really critical part of this. So we're not really like, oh, AI solves everything type
Type positioning. Although we definitely see use cases for it.
And Assistive Labs has been around for a couple of years now. I think we were the first assistive technology.
Would you put like screen readers and other test environments up in the cloud Came in about 2020.
And we've been working on this thing called accessibility focused or first end-to-end testing that is involving that.
Assistive technology cloud. And we also do some professional services enabled by that platform.
And very briefly, we have two products in the market. Live testing is what you see if you go to our website right now, the idea is that you can just load up in your browser and VDA or jaws or narrator or Windows icon contrast mode without needing to be on a PC or have that device ready.
Try to make that super simple. We'll be talking about this today. We have forgive the algorithm so a11y, for those who aren't familiar, is a numerism numerism for accessibility.
And then E2A is a shortening for end-to-end, but we have a technology that involves assistive technologies. This is the screen near NVDA's icon.
Browsers, scanners like Axe, Machine Vision, all that cool stuff. It involves automating it.
We also offer what we call evergreen audits. And the idea here is we do WCAG audits.
They're powered by this technology and platform. Complemented by human judgment and our services integrated inside CICD. We're excited about that.
That's all I'll say on it for now. Um so general context, the way we look at the space that we operate in and of digital accessibility.
A lot of companies are prioritizing accessibility. There are a number of different drivers from that, from grassroots stuff in terms of people wanting to bring up accessible design kind of from the bottom up, applying to brand values, regulations, all good stuff that we used to
Prioritizing accessibility. And when companies do prioritize it We usually see it turn into a really sustained multi-year effort. Accessibility is a job.
That's never done. So companies are thinking about their accessibility strategy and their maturity over time and kind of how to grow And we see a number of activities.
Done around that internally. Folks will focus on internal education. That's really common helping people at the company understand that accessibility is a priority trying to ramp up folks on the basics and more advanced stuff in all disciplines, design, PM, engineering.
Qa. Manual audits has always been a large part of it, whether internal and external or external, but working with experts in accessibility to kind of get an idea of a website or a product or an app's quality at any state and time, what are the issues in it.
We're big fans of user research. So testing your website, getting feedback from disabled users and the disabled community.
A lot of that going on. Lost checklist is a really shorthand way of talking about process, but the idea of when you are developing features, when you're doing work, accessibility is kind of part of your definition of done.
It's built into specs. It's considered at multiple parts in the process.
Shift left is a big part of this movement. And then the last piece here is I'm sure there's much more than this, but here are some of the big ones is acts and scanners, this idea of applying accessibility specific automation
To… your accessibility strategy. So we're going to focus in on that.
So yeah, companies are drawn to automation tools like X or scanners for a variety of different reasons.
For this talk, I'll hand it off to you, Nick.
A big one is that they're really fast and scalable. So you can scan hundreds of different pages on your website or application in you know a few seconds or minutes.
Which, if you were to try to manually go through all of those pages every time you scanned It would take a lot longer. You can also monitor your state of accessibility with some kind of historical record basically getting Some metrics and data as you're changing your website.
You can see how your act scans number of issues and things are changing over time.
And you're able to shift a little bit left by stopping bugs before they're shipped. If you have your axe scans or any kind of automation tools in your CI, CD pipeline, you can catch them.
Before they're shipped to production and users are interacting with them.
And you're able to kind of protect what's accessible by having these automatic tools preventing you from regressing or introducing new bugs.
And they're a lot cheaper usually than if you were to manually audit as much.
So a big common use case for automation is static code analysis.
This is also known as scanners. And you've probably heard of these ones like Axe Core, Wave.
Microsoft Accessibility Insights, which is really kind of just acts under the hood, Google Lighthouse, which is also really just acts under the hood with some different rules turned on and off.
And these are really great because they can find a lot of basic issues that you need to address. Say you have an image on your website that's missing alternative text. This is going to scan your website, look at every image and make sure that it has
Proper alternative text to make sure that you have accessible names on interactive elements.
That's really great. It tests on the rendered web page. So this is the DOM or the accessibility tree that you see. So when you go to a website, it's rendering it and then the scanner is scanning based on that.
An issue is that they're a little bit context blind so that they are just going to be the same kind of the same kind of scan running on every single part in your website and maybe there are areas in your website you want to focus in on certain things a bit more. It's not going to know about that. It's just going to
Give you an output based on what generated from the scan.
And it's unable to test interactions. So if you have Anything that's involved based on like clicking, you have a bug that's introduced from activating an element or moving something around, X isn't going to catch that axis just getting a static scan of the page at a moment in time.
Moving forward from that is flow analysis, which is kind of like scanners plus plus.
You get a bit more than that static moment in time because you're scanning had a bunch of different moments in time. So say you scan the homepage and then you scan the page Again, right after you open dialogue or you click a button that
Pops open a dropdown. Now you're getting more data. You're getting data for what it looks like in these two different states.
So flow analysis is basically you're taking a flow of like the user is filling out this form and opening these dialogues and you're scanning at a bunch of different points in time to get some more Just kind of data and scans in there.
So it's still context blind a little bit because it's the same general concept as a regular static code analysis.
Axe core scan or something. But you're just getting a lot more data put into it.
Drawn out of it. And a benefit here, though, is that it can find some bugs based on interaction. So say that you open up a dropdown And now that dropdown has some inaccessible, like it's missing an accessible name or something and one of the items
That's going to be caught now when it wouldn't have been caught before.
And then what about end-to-end testing? So that's something That is another type of automation. But you don't really hear about it as much in the accessibility space.
What can that do in terms of just generally in terms of just generally for when you're writing tests and you're trying to automate your development process.
So you've probably heard about end-to-end testing in some of these common examples like selenium, cypress playwright and puppeteer. These are all kind of drivers that will simulate a user going through actions on a website.
And yeah, these are all pretty powerful tools for end-to-end testing. So end-to-end testing basically mimics a real user interacting with the website.
So it'll load up some website it'll click on buttons to activate things. It'll do what you tell it to do. So if you're trying to write an end-to-end test that's checking that a user is able to sign up for a free trial. It's going to go to your website. You're going to make sure that you can
Click the Start Free Trial button. You're going to make sure that you can fill out your email And get an email and do all that.
And it's going to pass if you can do all of those steps from start to end.
So that's really good for stopping for regressions because it's going to fail if you introduce something that means that that flow can no longer be accomplished say you introduce a bug that means this this button that you click to start free trial, it doesn't open the free trial modal anymore. That's going to be caught now because end-to-end testing is going to say, oh, I can't complete this flow. I can't complete this
This test, so that's an issue. So that's really useful. Acts can find regressions too, you know, like a scanner can find regressions.
But this can find some like interactive regressions that are really useful and not able to really be caught from scanners before.
And it's primarily used for regressions because you're not going to write an end-to-end test when you're creating like a new feature Maybe if you're doing like test-driven development, you're going to try to have a test done before I can ship that along with the new feature. But if you have a new feature
And you want to check that that's accessible. It's not going to be really through end testing as much or check that it's working. You're going to do some just manual or some kind of analysis on the feature as you're creating it. So end-to-end testing is more about the regressions
And keeping things from breaking in the future. And it falls into the category of automated, not automatic testing.
So talking a bit more about that. Automatic testing. These are tools that work by themselves out of the box. So a scanner, a linter.
Something like Axe. So this is built-in self-acting behavior so you can just run it immediately and you don't have to do any additional work really. So you can say go to your homepage and just press them.
The scan button and you'll get a bunch of data. You'll get results about how accessible your page is or something.
An example of that is an automatic act scan found X, Y, Z, these number of issues on the home page.
Automated on the counter end of that is a tool that's set up to do a specific task when it's triggered.
So it's been configured by someone to validate specific criteria. So if you have an automated test that's going through a checklist of things. You want to make sure that this is here. You want to make sure that you can do this.
You want to make sure that when you go to this point, this button has this state.
An example of that would be an automated run from CICD revealed that the checkout flow is broken.
So this is a really crucial distinction. In understanding kind of the difference between the different types of automation tools.
And we have a bit of a demo right here to showcase that.
Yeah, I'll show that and kind of… familiar terms with accessibility.
So if I use our homepage as an example, and we focus on like the automatic piece of it, which these aren't like official industry terms. We just find it useful in uh talking about automation in terms of accessibility because i think
So many of us have just been accustomed to automation, meaning one thing, which is, oh, it's something like axe like I can install a Chrome extension.
I can click one button. All I had to do really was give it a web page and it did some stuff and figured out If I have any issues on this webpage.
Yay. But the difference or the distinction we're trying to make with automate automated is when you are putting more work into having the computer do that work.
For you. So if on this page I wanted to make sure our navigation worked, I want to make sure I can like open a dropdown and actually reach a landing page. Very simple scenario. You can imagine more complex ones with signing in, checking out.
But just to show you an example of how you would go about doing that, I'll show it with a tool.
Called Playwright, which was one of the ones we showed at the start.
And there's some bootstrapping stuff of working with these, but they are generally in code. There are some no code solutions you have.
Can have, but it's the same idea, like whether you're writing code or whether you're using kind of a form to create it.
You would start by writing down what you what you need to happen.
With playwrights.
You have this magical test keyword. And then you start declaring like, oh, I want to do a test of global navigation.
And then you start specifying what you want to do inside that test.
This is funky JavaScript syntax. For anyone, anyone not familiar, but I'm just trying to go through the high level of it.
So you would start writing down what you did. In manually.
So the first thing you need to do is probably load the page.
Playwrite has methods for that. I want to go to assistivelabs.com.
I want to click this dropdown here. And you need to tell the test somehow, like, what is that thing? And it supports a variety of methods that traditional historical way to do this is just like, oh, grab a CSS selector
To that thing. So you could say await page dot You can pass it some CSS.
So click that thing. And then when this dropdown's open, we also need to click the next thing inside it. So you can go ahead and grab that.
Selector as well. There's recorder tools to make this a little easier but I'm just spelling out the details.
Oops, forgot the crucial part of it. So you're clicking that thing.
Yes, I got the click over there. And then you can… assert on a end result. You might say like well I expect a page.
Url.
Let's see, when we go here. Url should be something like And accessibility testing.
So once you have that. Test written, that was all manual, nothing automatic about it.
But then you can automatically run all those steps in a repeatable fashion.
I could say example one. And this is going to be really fast.
Up the web page and it passes. That was a little too fast to see what was going on.
We can use.
We'll slow that down a little bit so we can watch what's going on.
The page loads. You can't see the mouse that was emulated, but it did click.
And expand the dropdown and it clicked again. It got to that final page and then it said, okay, we're done.
So this is a great example of a traditional end-to-end test, how they've worked for the past 20 years or more.
And sort of that automated fashion of it. I manually wrote it, but now I can run this as many times as I need to perform the same steps.
Yeah, and that is the benefit is that that is the Weston just talked about is really incredible that you can repeat that flow that you've just written now And it took like three seconds there, but that was running in head hit. It could run, I think, even faster sometimes
You can put that in your CICD checks and then every time you make a change, you can see, oh, am I able to still do those things that I was expecting to be able to do before?
Which helps you detect an issue before it's shipped. It helps you get this feedback loop so devs can see if they have broken something maybe and, you know.
Upload changes until the changes checks are all green. So it makes a really easy way for catching very specific bugs that are context like the context for your page is baked into them.
Into the tests.
So that's the definition of end-to-end testing and what it means in a traditional sense. If your team or company is using any of these tools, that's probably roughly how they're using them.
But there have been some strides to try to apply accessibility concerns into the end-to-end testing space, which is awesome.
But to start out as a baseline, just to appreciate In those tests.
These click methods are all about emulated mouse behavior. Foundationally, all these tools were written around the idea of we're emulating a user with a mouse.
On a desktop website. End-to-end testing can do mobile as well as the emulate touch. But there's some ableism baked into that.
That's been in the industry for a long time that was the primary goal of the primary goal of But that was the primary persona that the creators of end-to-end tests frameworks had in mind was, okay, somebody with a mouse who can see.
So that's important to appreciate as kind of the perspective it's coming from.
Not all end-to-end testing frameworks can accurately interact with the keyboard at all. Some can, but the support's super rudimentary. The things you would kind of need to do to interact with keyboard or hard to write out.
Um and these things over here, they're called locators or selectors. That's a really core problem in end-to-end testing. How do you write down the thing to go click on is.
And for the longest time. Accessibility wasn't really any factor inside that at all.
There was basically two checks that And then test the frameworks would do they would Make sure, okay, is a thing on the page that matches this visible And is it able to be clicked? Does it make sure it doesn't have like a dialogue over it or something? If both those things are true.
Like, good to go. Keep going. And of course, that doesn't account for a huge amount of accessibility things. You don't know if it can be keyboarded to. You don't know if it can be interact with a screen reader. You don't know if it has an accessible name or role.
So if you try to map mental model of coverage out of this. And we'll reuse this Venn diagram several times throughout the talk.
If you pretend that a human, a skilled human could potentially find any accessibility issue present on a website.
They can find it on the initial page state, on any interaction, like they could find anything in theory.
Traditional end-to-end testing. It can do a very small fraction of those things um you know there's aspects of WCAG that talk about pointer gestures. And it's like, yeah, you get coverage of some of those things.
But a really tiny droplet in the bucket. And by the way, these Venn diagrams are not to scale. We didn't calculate out numbers. This is just to to kind of put things relative to each other.
But luckily there's been some advancements in the field. One that is semi-recent, like past five years is this idea that instead of locating elements by the CSS selector or something else.
We're actually going to try to use their accessibility information. So there's these utilities like get by role is the idea that you can say Hey, I want a button with the accessible name of use cases.
And go find that on the page for me. And that adds on to add on.
Traditional checks of just visible and clickable and also make sure that like, yeah, this needs to be a button in the accessibility tree.
Which is great. And that does increase coverage a little bit. The test will fail if there is no longer a role of button if the switches to link or loses its role altogether, the test would fail.
But it's really tightly scoped inside like just the things that you're clicking on.
And then the second advancement Nick talked about a bit before that fits into the scanners plus plus category is the idea that it's super easy to take a large amount of existing end-to-end tests written in selenium or cypress or playwright
And do an integration with an accessibility scanner like Axe or Events or Wave.
And they make this really simple. It's like one line of code to put it in.
And basically, the integration will run the scan on every page when it loads and then it also will try to run after interaction. So just try to get a whole bunch of data into these scanners And there's a lot of complexity with that.
Hard to pull off sometimes but sometimes Overall, it's a good thing like because these scanners can check a whole bunch of kind of low hanging fruit. You're getting a whole bunch more coverage the more data you feed it, kind of the better, as long as you can
Can handle and dedupe that data. So we think this is pretty representative of the end-to-end testing technology available today, at least in the mainstream. There's some other efforts But we think these are the main two.
Things you'll probably see out there. But we asked ourselves, how much further can it go? What would it look like to really apply an accessibility lens?
To end-to-end testing. Um so we came up with this idea of accessibility first end-to-end testing. I don't know if we came up with it. A lot of people have thought about this problem. But the way we're defining it is take this practice of writing test in automation where you're writing it for a specific website, a specific product. You're trying to do it from the end user's perspective.
But consider accessibility from the ground up, from everything you're building.
It needs to be able to handle keyboarding, screen readers, WCAG validations.
All that good stuff. What would that look like? And we realized kind of the raw materials that you need in a framework like that are different.
From traditional end-to-end testing, which is pretty much Just like, oh, we need access to the browser and we need a mouse that we can emulate.
You need access to the browser still, but you also want really good control of the keyboard.
You do still want scanners inside there. They're wonderful and they are very broad. You probably want access to screen readers to check compatibility inside that.
You want access to machine vision tools because there's a lot of aspects of accessibility that are visual based.
Like, did something get bigger? Does it have the right amount of contrast? Is it visible or invisible?
And then there's this whole kind of toolbox of things that you need to to work with to do accessibility validations.
So I'll hop into a demo of a demo what we've been been working on in this space and kind of our take on how you might work with that.
So coming back to this example of the homepage. If I were to take that same sort of scenario of like, well, I want to make sure that our dropdown is working really small, but nicely self-contained to fit it in the space of an hour demo.
If I were to manually accessibility audit this, my first step honestly would probably be opening the page and running axe on it and just seeing if that that detects anything.
After I'd done that, I'd start probably using keyboard a little bit and interacting with the page, making sure I can use tab to get to this dropdown, making sure I can open it.
With the keyboard and that focus is managed correctly I'd also start probably opening the developer tools, getting curious of like, well.
Okay, what is this? All right, that's interesting you know we put in a banner landmark. That's important.
For multiple aspects of accessibility. We also have a navigation landmark inside there.
That's helpful.
And… I'd be looking at contrast. I'd be looking at whether focus is visible here. And I'd probably come back and do some validation with a screen reader.
So I'll pull up in Assistive Labs, you can pull up nvda Since I'm on a Mac here.
But if I go through that kind of same flow with NVDA up.
I'll start paying attention to what its announcements are and what that experience is like.
Assistivelabs.com Selected, Assistive Labs document.
So that's interesting. Assista Bob's document. That's my announcement of page title.
I think about that in terms of when is the webpage ready for interaction?
Also page title is accurate and good to have for the homepage.
And then I'll start. Performing the same interactions.
So that's good. Nvda announced that we're inside the banner landmark.
Banner Landmark, skip to content link.
Skip link is great that that's present.
It's good that we entered the navigation landmark. Notice it said current page for that link. So we're using uh I think that's area current property to help cue people in as to where they are on the website navigation.
Navigation landmark, AssistiveLabs link visited current page.
So semantics there. It's saying, okay, you've entered a list. This is a button. If there's state.
List, use cases button collapsed.
The buttons currently collapsed. You're informed of its state change it's expanded.
Space, expanded.
List, end-to-end accessibility testing link. End-to-end accessibility testing, assistive labs, end-to-end accessibility testing, assistivelabs document.
And for better or worse, we're a single page app. So that wasn't a normal page transition. So focus was managed. I went to the H1 on the page, so you'd have a notification that the new page is ready, what page you're on.
And I'd probably run Axe again on this page, but those are just some of the checks I would do.
So how would I go about doing that? In an end-to-end testing fashion.
So let's create a second example here.
So instead of working with playwright, we're working with a test framework uh that we've developed, but you work with it in a very similar way you're declaring what a test.
Is and then instead of getting a browser page, you're getting access to this assisted page.
Which is like a browser, but you also have access to a bunch of other stuff.
So the first step we need to do is very similar. We've got to load that page.
But then the second argument uh it's this function here is talking about speech.
So it's saying, while speech is optional like optional consider passing what a screen reader user would rely on to know the page is loaded.
So this has to do with 4.1.3 status messages. And also just general user experience. And this is also a chance to capture the document title.
So we want to make sure NDDA loads the page and says that announcement. And then we want to do a scan of the page.
So easy access to using Axe inside it. And after that, we want to start thinking about what we're interacting with.
So I started tabbing and the first thing I hit was a skip link.
So I'm going to start declaring in accessible terms what I need to interact with. That's in the dropdown.
So we work with us in the concept of targets and targets are very accessibility focused they start by thinking about accessible name.
Which I think was skipped to content. You must specify an accessible role, which in this case is a link.
Doesn't have a value. And then… things have visible labels. And in this case, it's the same thing.
But that's an important aspect for like 2.5.3 label and name voice control users Generally, you want to establish this link between accessible names and what's actually visible.
On screen and it would be a problem if they diverged because then, for example, if this changed to Just say… jump to content, for example. Now, a voice control user wouldn't be able to invoke it because they would actually need to say skip to content.
We need to interact with that skip link.
It'll make it really easy to bring focus to it with a keyboard.
This comes with a lot of defaults. It knows that tab is typically the way you interact.
I'm also going to tell it like this should be the first link on the page. So just press tab once and it should be focused.
And I believe I also need to tell it. You have to press tab and just trust it because skip links are invisible initially.
And then the next thing we interact with is like the homepage here. I'm going to start pulling some some code that I'd written beforehand to save time So in sort of a very similar way here we have our homepage link. In this case, it doesn't have a visible label. It has an icon.
And we would actually want to fail the test if it started having text inside it because That could potentially mean that it's accessible name in the text diverged.
But you want to focus that. And you also want to capture some extra information from screen reader like NVIDIA, you want to make sure that NVIDIA is announcing the navigation landmark And it's announcing that… link is the current page.
So let's stop there and just get this basic test up and running and see how it works.
So just like before, it pops up in a browser But it's a little confusing because there's a purple screen and then another screen.
Where you are connected to a remote test environment. And this is the Assistive Labs Cloud.
This is a remote machine that has a browser, has NDDA up and running on it.
Desktop list. Chromium.
That serves as a environment that you can automate.
Assistive labs, Chromium. Assistivelabs document. Sleep mode on.
And this is exposing a funny responsive design bug there. But right now it's just scanning the page with Docs and actually closed because it found some issues.
Desktop list. Recycle bin not selected.
And so it's saying, hey, we actually detected that there's a label content name mismatch, which is interesting because I ran the Axe browser extension.
And I didn't find any issues. This is complaining that this chat button down here.
Has the accessible name as
I think it says help scout beacon open But visibly it's chat.
So an end-to-end testing, you have much more control over how you use Axe. You can enable experimental rules, you can enable kind of everything you need. So that's a way to get more out of tools like Axe.
So that's interesting. And then it's also interesting talking about things that it's not entirely sure are an issue, but in this case, it's saying, hey, the YouTube player down the page is a div.
That has an area label on it and area label on div isn't really well supported because there's no role associated with it.
So heads up on that. So you can get more power from your axe scans this way.
And you can capture all that stuff. So I actually have some stuff ready to go here.
So there's some code you can copy and kind of snapshot on the page of like, here's what I expect from the scan.
Inside here. In this case, I'm going to capture two things.
One is, yes, I know about the label content. Name mismatch on that help scout thing. I actually probably want to track this inside our issue tracker. So I would open a link inside our an issue inside our GitHub. Then it's going to live inside this test and get revalidated.
Every time the test is run, which is a cool aspect of accessibility because there's usually always issues hanging around And then I'm also tracking the area label in this case.
I'm going to probably ignore that. Youtube has its own its own deal for better or worse. We're just going to accept its faults.
So you can start tracking that kind of stuff in your scan.
And let's run this test one more time.
Things slowed down quite a bit so we can observe more of what's going on.
Desktop list. Chromium.
Assistive labs, Chromium. Assistive labs document. Sleep mode on.
Sleep mode off. Better landmark. Skip to content link.
So it's performing those interactions. Nvidia is announcing Let's figure out what I did wrong.
Recycle bin not selected one of
It's telling me that, hey, did you know that this is actually announcing as a banner landmark?
And that's cool. I care about that.
So you keep moving in this fashion and you can create a large test of of the page and I'll jump to the pre-written example here Where?
Clear this out a little bit. You're getting to the… the home link, you are getting to the use cases button, focusing that button, you're invoking it with keyboard. You're making sure that it's expanded states announced successfully. You do another scan because you want to check the page state once that
That dropdown's open. Then you want to access the link inside the dropdown with keyboard and the screen reader And you want to make sure that if you invoke that link, you make it to the next pages H1 correctly, and then you want to run a scan on that page.
So very briefly. We'll zoom through a video of that.
All that gets automated into automated into a robotic run that's performing those same steps. So it's opening the dropdown, performing scans, getting to that final state, scanning again.
And passing because it's able to perform all those steps and it didn't detect any issues.
So what kinds of things can you find with this when you think of the difference it makes in terms of coverage. It's definitely good to know that that drop down works.
It makes sense that this test proves you can get to it with keyboard.
And it also proves a lot of things are announced by NVDA that you expect when the test is passed.
But a couple examples of bugs it confines that tools like Axe can't.
Are… If you have a website.
That has the… the doomed outline none bug that a designer has added. So I'm pressing tab right now and the skip link showed but Really, I can't see visible focus because outlines have been hidden.
Acts. Would pass on this page.
The automated test detects the elements focus state is invisible.
You can look at a report of that and see, hey.
When we tried to focus the the home page, it's focused and unfocused states are exactly the same so that's a failure. And you can work around that failure and get past it but the test flags that for your review so you can open an issue.
Another example here. Let's say you have a focus management issue so I can get to the dropdown, I can expand it with keyboard. But then when I press tab.
Focus doesn't go into the dropdown. It accidentally goes past it.
This isn't something you would catch with mouse end-to-end testing. You would just click the link inside here and it would be fine.
But again, you catch it because the test is saying, hey.
I used keyboard. I press tab and expected to get to this link inside the dropdown.
But instead, I found my focus over on this other button.
It's really powerful in that regard. And one final example here.
In this case, it's saying, hey, when I got to this uh element and i pressed space to expand it.
Nba didn't say anything and actually didn't open on the page, but it's saying, I expected to hear expanded from the screen reader, but I got nothing.
Instead, I got a renouncement at the bottom that it was collapsed.
So those are three examples of the types of issues you can find. There's far, far more from that.
But to tie that back to that kind of a larger picture of of coverage We think that we think relatively significantly expands the amount of stuff that you can find with a test because unlike scanners like axe, you're able to cover interactions. You're able to actually test screen reader
Compatibility. You're able to assess a lot of the visual things that aren't able to be assessed.
In a scanner context. And to put some numbers to this.
For better or worse. I think it's important to try to quantify this stuff. It's also really tricky.
But if you ask the question for WCAG 2.2, level A and level AA success criterion SCs. There's 55 of them.
And if you just ask the question of if a tool has a rule that pertains in any way to one of those success criteria, so touches it even very lightly.
So for example, we would say axe touches 2.1.1 keyboarding because it'll validate that you can tap to iframe. So you can scroll them with keyboard.
That's a tiny, tiny fraction of your concerns for keyboarding, but we would count that.
If you tally that up, Axe can do 40% of that.
55 today. And end to end can do with Axe Stenny on top of it 80%. So you're doubling your coverage in a kind of different way though.
And by the way, this is including Axe's experimental rules. So like everything you could possibly kind of configure acts to be able to do.
But the important distinction there is when you're thinking of automatic tools like Axe.
These are generally high breadth tools. They're looking at the whole page. They excel at that.
Top to bottom, your footer everything. But they do tend to be relatively low depth depending on the area because they don't understand that context. They don't know how to open your dropdowns in the the global navigation. They don't know how to sign in or do anything, fill out forms
Whereas automated tests with end-to-end They're built with automatic tools, so you retain high breath But the depth really varies.
If we zoom in here. You have the same idea where you're getting full page coverage because you are doing those scans, but then that extra 40% of coverage is going to be limited to where you're heavily interacting with. So you're really positive
That the global navigation is working well in this case because you're checking a whole bunch of aspects of it. You're less positive about the footer or other aspects of the page.
It's really up to you. The more end-to-end tests you write, the higher that depth of coverage goes. The less you write, the lower that of coverage goes.
So kind of tying it back to those automation benefits from before. End to end is really interesting because it provides high coverage as compared to the existing tools.
That are commonly used today. It retains that ability to be fast and scalable.
And it retains that ability to help you monitor with the caveat that it can do it better.
When you have that higher coverage, you have a much clearer picture of your accessibility at any point in time.
If you're just running X, you're just seeing a part of that picture.
You know imagine trusting that your website's accessible without validating any of the interactions to it. That sounds kind of scary.
And it's really awesome at stopping backsliding and regressions. Again.
It's high coverage. So the higher the coverage, the more aggressions it can find, the more it can help you protect all the things that you put work into making accessible.
Notice I didn't put cheap as the last thing inside here.
This is more effort than… acts and like those automatic tools that are pretty straightforward to integrate.
One line of code to get it in place and end-to-end test suite or one click to run a scan on a page.
These are people who are skilled in accessibility writing tests and building up this library of tests. And that takes effort and that's Definitely an order of magnitude greater than working with tools like Axe.
But tying that back to some of what we do. I think this technology, this approach can be generalized a good bit.
Don't necessarily need our platform and framework to do it. But one way we try to support people with that is by helping them write those tests. So that's the idea that You work with our team like uh an auditor.
And we do an audit, but we also build a test suite of that audit that integrates into CICD.
So you're getting this really cool kind of We think of it as evergreen validation of an audit scope. So if you really care about your P0 features, if you really care about your checkout experience, that's the type of stuff you can
Create an end-to-end test suite over and we help you do that.
And manage all the results from it. And so we kind of work with you in a white glove way on that.
Cooperating on an audit scope. But it's very different from the idea that you're going to run acts on every page possible on your website. Usually you create end-to-end test suite around 25, 50 use cases.
And you prioritize your highest. Highest traffic and highest risk cases.
We have a case study out with Asana that I'd love if folks checked out, but they were really excited about how this kind of changed things, just the dramatic difference they felt from getting high coverage accessibility validation in CICD.
And… 10 minutes left. I will stop there for questions.
Nice. Good job. Yeah, great timing too. Plenty of time for questions if people have them.
Thank you all.
There was a question about focus visible, you know, detecting that and you covered it kind of right after the question was asked.
Is there anything else that people are curious about like what does this um Or what's your reaction to it? What do you think? Have you seen anything like this before?
I can add in. So, I mean, there's some existing tools that go beyond like the traditional end-to-end automation on the market um One is totally free called Guide up.
Okay, so…
And that's in the web accessibility slack it's bounced around in there a bunch. And it's really cool.
It's a way that you can automatically drive NVDA or voiceover.
I think those are the two main ones through code, which is kind of a similar idea.
And encourage folks to kind of check that out and play with it.
Our take on it is. Yes and like you want that and then you want a whole bunch of other stuff on top of that. You want the machine vision pieces. You want heavy keyboard validation. You want to treat.
Keyboarding and screen readers as as both concerns that you need to validate.
And all that good stuff. So this is one of the cooler projects we've seen.
Out in this space. There's also been some cool blog posts that we can link to, I think Mark Stedman had some cool stuff around not having access to screen readers, not having access to machine vision, some of the stuff you can do in traditional end-to-end tests about asserting that like
Did the dialogue open? Where is focus right now? Even if you are using quick interactions, you can still check things like is the right area attribute present.
Jen, did you have a question? Saw your hand up there for a second.
Sorry, I was struggling with my… Mike there. It was really just a comment i I really enjoyed seeing your framing of the end to end.
No worries.
To your earlier point, so often I see people and teams stopping with acts and not understanding that there are more nuanced interactions to be assessed.
So to see something that doesn't feel so daunting is great to see. So thank you so much for sharing this.
I've seen that effect in a number of places where teams will set up acts And some of the results are really self-explanatory, like missing alt text and people fix them. But sometimes acts will generate results that are confusing to people who don't have accessibility expertise.
Anx doesn't frame its results really in a usability context. Again, because it doesn't understand your page.
You know, it'll say that air about the YouTube player. It'll say, hey, you've got a div here with an area label.
What does that actually mean? Like, what does that mean to a screen reader user? Does that mean that someone using NVDA, can't interact with the YouTube player.
So teams will struggle with that and kind of have one or two actions you can put a bunch of effort into just trying to quiet Axe down and like, I'll appease you Or you can end up ignoring it and just saying, well, I don't understand this so
I'm just going to leave axon in the background and that doesn't it kind of turns into a source of noise, which is a rough state to end up in.
Definitely. Lots of friction and just so much interpretation needed to be had.
Especially without having access to people who rely on these forms of assistive technology so lots of gratitude here.
That's something that a sauna case study. That's one thing we really enjoy doing in our evergreen audit service is we act as a human in the loop.
For that stuff. So when Axe does generate failures, we don't pass that on to teams we write up a professional issue with that that explains it in very like beginner friendly terms.
Which is useful for acts, really useful for all the stuff that Acts can't cover too.
Focus management changes in screen reader announcements. Is it a problem that Axe no longer announces banner landmark?
Depends, right? There could be a lot of reasons.
Cool. Anything else?
Right. This has been great. Thanks. I mean, great job, Weston. Great job, Nick.
Thank you all for being here and for your participation and um Yeah, like I said, I'll be posting video, the recording of this soon, and we'll get audio descriptions added to that as well.
But yeah, if this has kind of piqued your interest and you want to learn more, you want to dive deeper with us, get in touch. We're happy to get on a call and kind of walk you through even more details about it.
But yeah, thanks so much for being here. Take care, everybody.