Decentralized ASR Correction Using On-Device Feedback Loops
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice-based user interfaces face inaccuracies in speech recognition, particularly on-device models, due to resource constraints and limited training data, leading to misrecognitions that are difficult to correct without compromising user privacy.
Innovation Solution
A decentralized approach where on-device processors capture user corrections to predicted text segments, store them locally, and transmit candidate correction pairs to a remote system for verification across multiple devices, updating a global ASR model based on verified corrections to improve recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech recognition is performed on-device, then data security and network usage are improved, but recognition accuracy deteriorates due to resource constraints and limited training data
Solution Approach 1:
The system segments the speech recognition task into two parts: local execution for privacy and speed, and centralized learning for accuracy improvement. The on-device ASR model handles recognition locally, while correction data is aggregated and processed centrally to update the model without compromising local privacy or resources.
Solution Approach 2:
The system implements a feedback loop where user corrections to ASR predictions are collected on-device, aggregated, and used to update the centralized ASR model. This feedback mechanism allows continuous improvement of recognition accuracy without requiring users to manually retrain models or expose their data.
2Measurement precision
If additional speech recognition hypotheses are generated to recognize new terms, then recognition accuracy is improved, but device complexity and processing requirements increase
Solution Approach 1:
The system extracts the complexity of hypothesis generation and model training from the on-device environment. Instead of running complex post-processing and multiple hypothesis generation locally, the device simply collects corrections and transmits them centrally, where the computational burden is performed on powerful cloud servers.
Solution Approach 2:
The centralized server acts as an intermediary that handles the computationally intensive tasks of generating alternative hypotheses and training updated models. The on-device system only performs simple data collection and transmission, eliminating the need for complex local processing while still achieving improved recognition accuracy.
3Measurement precision
If user corrections to speech recognition text are collected to improve accuracy, then recognition accuracy is improved, but user privacy and data security are compromised
Solution Approach 1:
The system extracts only the necessary correction information (the difference between predicted and actual text) from the user's input, discarding all other audio data and personal information. This selective extraction minimizes the data transmitted and stored, reducing privacy risks while retaining the useful correction signals for model improvement.
Solution Approach 2:
The system applies different quality standards to different data: audio data remains local and untransmitted, correction metadata is aggregated and transmitted only in anonymized form, and model updates are applied centrally. This differentiated approach ensures that high-value privacy-sensitive data stays local while still enabling accuracy improvement through centralized processing.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Implementations described herein identify and correct automatic speech recognition (ASR) misrecognitions. For example, on-device processor(s) of a client device may generate a predicted textual segment that is predicted to correspond to spoken utterance of a user of the client device, and may receive further input that modifies the predicted textual segment to an alternate textual segment. Further, the on-device processor(s) may store these textual segments in on-device storage as a candidate correction pair, and transmit the candidate correction pair to a remote system. Moreover, remote processor(s) of the remote system may determine that the candidate correction pair is an actual correction pair, and may cause client devices to generate updates for a global ASR model for the candidate correction pair. Additionally, the remote processor(s) may distribute the global ASR model to the client devices and/or additional client devices.