Using Chrome Live Caption to augment speech viewers

Screen reader speech viewers are notoriously inaccurate. Luckily, Assistiv Labs’ live testing product supports Google Chrome Live Caption. Read on to learn when this is a viable way to ensure your testing results are correct.

The problems with speech viewers

In our live testing platform, we enable NVDA’s speech viewer by default. It’s a helpful reference for when you’re not quite sure what NVDA just said or missed some words. Some users prefer to keep NVDA muted and rely entirely on the speech viewer. And hard of hearing or deaf users rely on speech viewers in lieu of captions.

That’s great! Except, as Adrian Roselli says in his article, Speech Viewer Logs of Lies:

The problem is that what you see in those speech viewers is not always what you hear.

Another problem is that neither JAWS, the most popular screen reader, nor Narrator, the screen reader built into Microsoft Windows, even offer equivalent features to NVDA’s speech viewer.

To summarize, not all screen readers have speech viewers and even when they do, they lie. #SadTrombone

Live captions to the rescue?

Fortunately, Google Chrome has for a few years now had the ability to live caption audio content on webpages. Because this feature attempts to caption all audio output from the browser, it works with NVDA, JAWS, and Narrator being run in an Assistiv Labs testing session.

The gory (technical) details

Live Caption hasn’t always worked with Assistiv Labs sessions. As of version 115 released July 2023, Live Caption in Google Chrome started supporting audio sources other than the HTML5 <audio> and <video> tags. Similar to apps such as Zoom, we use the Web Audio API to deliver audio to your browser, so before that version of Chrome, Live Caption didn’t work.

But now it does! #CueConfettiCannon

Cool, so…how?

To use Live Caption as part of a testing session, follow these steps:

1. In Chrome on your machine (i.e., not Chrome inside the Assistiv Labs testing session), go to Settings > Accessibility (i.e., chrome://settings/accessibility) and enable Live Caption (if this is the first time you’re enabling the feature, it may take up to a few minutes to download the required language files from Google).
   Live Caption is conveniently located at the top of Google Chrome’s Accessibility settings.

2. Start a new testing session using Chrome.
   Remember, you need to enable Live Caption in the Chrome on your system, not the one inside your Assistiv Labs session.

Caveat emptor

Live captioning is never perfect, and Chrome’s Live Caption feature is no exception. The following are some of the caveats we’re aware of for accessibility testers, in particular:

• While symbols and emoji are no longer breezed over, (i.e., you’ll see “dash” in the live caption instead of “-” in the speech viewer if the dash was announced, and nothing if the dash wasn’t announced), the live captions sometimes look like gibberish and may not produce consistent results.
• Live Caption struggles with jargon since it doesn’t have access to the actual text of the page (e.g., proper names, technical terms, URLs, etc.).
• It doesn’t capture earcons, like NVDA’s alert sound when switching between focus and browse modes.

Additionally, NVDA’s speech viewer can show a lot of not-announced text. This may be important context for testers who need to validate certain accessibility features. Live Caption only displays what is announced, so this can be useful for understanding the difference between what’s announced and not. The caveat here is that there is a slight delay between text being announced and it showing up in captions. Also, captions generated by Live Caption are ephemeral. They disappear after a few seconds.

Caveats considered, we believe Live Caption can be a handy tool for understanding what is actually being announced by the screen reader you’re testing.

Creating a great testing environment is something we care deeply about and are always looking for ways to improve. If you have ideas for how to make the testing experience better, let us know!