On-Device Recognition Training with Synthetic Speech from Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inaccurate speech recognition occurs frequently in on-device models due to resource constraints and lack of training data for new or infrequent terms, leading to inefficiencies in voice-based user interfaces.
Innovation Solution
An on-device speech recognition model is trained using locally stored textual segments through a speech synthesis process to generate gradients, which are used to update the model weights, improving recognition accuracy without requiring actual human utterances. This process can also transmit gradients to a remote system for updating a global model, conserving network resources and preserving privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If on-device speech recognition model is used, then device autonomy and privacy are improved, but recognition accuracy deteriorates due to resource constraints and limited training data
Solution Approach 1:
The system performs preliminary action by synthesizing speech data from textual segments and using it to pre-train the on-device speech recognition model before actual use. This allows the model to be adapted to device-specific characteristics and vocabulary in advance, improving recognition accuracy while maintaining device autonomy. The synthesized speech serves as preparatory training data that compensates for the lack of real human utterances.
Solution Approach 2:
Speech synthesis acts as an intermediary that bridges the gap between textual segments and real speech data. The system uses speech synthesis to generate artificial speech samples from text, which then serve as training data for the recognition model. This intermediary process enables the model to learn from text without requiring actual human recordings, resolving the contradiction between device autonomy and recognition accuracy.
2Adaptability or versatility
If additional hypotheses generation is performed, then recognition coverage for new terms is improved, but device resource consumption and processing time increase
Solution Approach 1:
The system performs preliminary action by pre-processing textual segments into synthesized speech and training the model in advance. This preparation work is done before runtime, so that during actual speech recognition, the model can quickly process queries without requiring complex additional hypotheses generation. The preliminary training enables the model to handle new terms efficiently during operation.
Solution Approach 2:
The system uses itself to generate training data by employing speech synthesis to create artificial speech samples from its own textual segment database. This self-service approach eliminates the need for external human recordings or complex data collection processes, allowing the model to improve its own performance using only device resources without increasing processing complexity during runtime.
3Measurement precision
If cloud-based speech recognition is used, then recognition accuracy is improved, but network dependency and latency increase
Solution Approach 1:
The system performs self-service by training and executing the speech recognition model entirely on-device without requiring cloud connectivity. The model learns from synthesized speech generated locally from textual segments, enabling accurate recognition to be performed autonomously on the device. This eliminates network dependency and reduces latency while maintaining recognition accuracy through local model adaptation.
Solution Approach 2:
The system performs preliminary training of the recognition model on-device using synthesized speech data before actual recognition tasks. This preliminary action prepares the model to operate independently without cloud assistance, reducing runtime latency while achieving cloud-level accuracy through advance local adaptation to device-specific characteristics.
4Measurement precision
If more training data is collected, then model accuracy is improved, but data privacy risks and storage requirements increase
Solution Approach 1:
Speech synthesis serves as an intermediary that transforms textual segments into synthetic speech data for training purposes. This eliminates the need to collect and store actual human speech recordings, as the system generates its own training data from text. The intermediary synthesis process improves model accuracy while avoiding privacy risks and reducing storage requirements associated with collecting real user data.
Solution Approach 2:
The system generates its own training data autonomously by synthesizing speech from its existing textual segment database, without requiring external data collection. This self-service approach improves model accuracy using only device-stored text, eliminating the need for additional data storage and avoiding privacy concerns associated with collecting sensitive user speech data.
Data Source
AI summary
Processor(s) of a client device can: identify a textual segment stored locally at the client device; process the textual segment, using a speech synthesis model stored locally at the client device, to generate synthesized speech audio data that includes synthesized speech of the identified textual segment; process the synthesized speech, using an on-device speech recognition model that is stored locally at the client device, to generate predicted output; and generate a gradient based on comparing the predicted output to ground truth output that corresponds to the textual segment. In some implementations, the generated gradient is used, by processor(s) of the client device, to update weights of the on-device speech recognition model. In some implementations, the generated gradient is additionally or alternatively transmitted to a remote system for use in remote updating of global weights of a global speech recognition model.