Web Speech API Contextual Biasing

[evanbliu]

WebKittens

@marcoscaceres

Title of the proposal

Web Speech API Contextual Biasing

URL to the spec

https://webaudio.github.io/web-speech-api/#speechreco-phrase

URL to the spec's repository

https://webaudio.github.io/web-speech-api/

Issue Tracker URL

No response

Explainer URL

No response

TAG Design Review URL

No response

Mozilla standards-positions issue URL

mozilla/standards-positions#1255

WebKit Bugzilla URL

No response

Radar URL

No response

Description

The Web Speech API's contextual biasing functionality allows developers to improve the accuracy of speech recognition by providing a list of specific words or phrases that are likely to be spoken. This is accomplished by creating a SpeechRecognitionPhraseList containing SpeechRecognitionPhrase objects, each with a text string and a numerical "boost" value to indicate how much more likely that phrase is. This enables the speech recognizer to be tailored to the specific context of a web application, such as one containing unique brand names or technical jargon. By giving the recognizer these hints, it is less likely to misinterpret these specific terms, leading to a more accurate and reliable user experience.

[marcoscaceres]

Although the use case seems interesting, the part linked to in the spec doesn't answer how the values of SpeechRecognitionPhrase "improve the accuracy of speech recognition by providing a list of specific words or phrases that are likely to be spoken."

On the design, introducing yet another list type (SpeechRecognitionPhraseList) seems to go against well established conventions (i.e., just use a WebIDL sequence... please don't create another list type)... yes, the original Web Speech API is already full of not so great things, but we shouldn't introduce more. That should be a sequence.

It seems somewhat unfortunate and concerning that these things are landing in the Web Speech spec without universal consensus from all implementers that they should be added to the spec. The spec should reflect reality of what is implemented, not have things in it that haven't been agreed to by all implementers.

[evanbliu]

I've drafted an explainer that contains a few example use cases of how speech can be biased using SpeechRecognitionPhrase. The boost value is somewhat arbitrary at the moment, but in the Chromium implementation, it approximates the natural log of how many times more likely you think a phrase is compared to what the recognizer knows beforehand.

Good point! I've opened an issue to replace SpeechRecognitionPhraseList with sequence.

We would love to get more input from Apple on the recent work in this area. I've heard that there might not be anyone at Apple actively working on the Web Speech API, but if you know of anyone, please let me know. I would be thrilled to include them in our discussions. The work on the Web Speech API is happening in the Audio Working Group, primarily with members from Google, Mozilla, and Microsoft.

[evanbliu]

FYI, there are some discussions on removing SpeechRecognitionPhraseList here: WebAudio/web-speech-api#169

[evanbliu]

The SpeechRecognitionPhraseList has been replaced with ObservableArray in the latest version of the spec. We're now requesting this to ship in Chromium. Thanks for your feedback!