On-Device Speech Model Gradient Updates from User Corrections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice-based user interfaces face challenges with inaccurate speech recognition, particularly for new or infrequent terms, and this issue is exacerbated when speech recognition is performed on-device due to resource constraints.
Innovation Solution
The method involves generating a gradient based on user corrections to predicted textual segments, which is used to update the weights of an on-device speech recognition model. This process improves speech recognition performance by adapting the model to the user's speech characteristics and frequent textual segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech recognition is performed on-device, then data security and network usage are improved, but speech recognition accuracy deteriorates due to resource constraints
Solution Approach 1:
The system enables the on-device speech recognition model to autonomously improve its own performance by generating training samples from user corrections and updating its weights locally. This self-service mechanism allows the model to adapt to user-specific speech patterns without requiring cloud-based processing, thereby maintaining data security while improving recognition accuracy over time.
Solution Approach 2:
The system dynamically adjusts the model's weight parameters based on gradient computations derived from user corrections. By changing these parameters locally on the device, the model adapts to improve accuracy for specific users while remaining on-device, resolving the contradiction between data security and recognition precision.
2Measurement precision
If additional speech recognition hypotheses are generated, then recognition of new terms is improved, but device complexity and processing resources increase
Solution Approach 1:
The system extracts only the essential corrections from user interactions and uses these extracted data points to generate training samples. By taking out only the necessary information rather than processing all possible hypotheses, the system improves recognition accuracy for new terms while minimizing the increase in device complexity and processing requirements.
3Measurement precision
If the on-device model is trained with user-specific data, then speech recognition performance for that user is improved, but model training time and computational resources increase
Solution Approach 1:
The system performs partial training by using only the corrections and feedback explicitly provided by the user as training samples, rather than attempting to process all possible speech data. This partial action approach enables the model to adapt to user-specific characteristics and improve accuracy while limiting the time and computational resources required for training.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Processor(s) of a client device can: receive audio data that captures a spoken utterance of a user of the client device; process, using an on-device speech recognition model, the audio data to generate a predicted textual segment that is a prediction of the spoken utterance; cause at least part of the predicted textual segment to be rendered (e.g., visually and/or audibly); receive further user interface input that is a correction of the predicted textual segment to an alternate textual segment; and generate a gradient based on comparing at least part of the predicted output to ground truth output that corresponds to the alternate textual segment. The gradient is used, by processor(s) of the client device, to update weights of the on-device speech recognition model and/or is transmitted to a remote system for use in remote updating of global weights of a global speech recognition model.