On-Device Speech Recognition With Multitask Punctuation Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech recognition technologies face challenges in deploying models on terminal devices due to their large size and require complex processing, and punctuation prediction accuracy is low, especially in long speech recognition scenarios.
Innovation Solution
A small-sized multitask neural network model is developed that can be deployed on terminals, capable of simultaneous character and punctuation prediction without requiring word segmentation, using a streaming or non-streaming network structure to improve prediction accuracy and reduce memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a conventional speech recognition model is deployed on a terminal device, then the model can perform speech-to-text conversion, but the model size becomes large and requires complex processing
Solution Approach 1:
The patent merges character prediction and punctuation prediction into a single multitask neural network model. This integration allows the model to perform both functions simultaneously with a unified architecture, reducing overall model complexity compared to using separate models for each task while maintaining comprehensive speech recognition capabilities.
Solution Approach 2:
The neural network model is designed as a universal multitask system that handles both character recognition and punctuation prediction. By making the model multi-functional, the patent eliminates the need for separate specialized models, thereby reducing device complexity and memory requirements while preserving reliable speech recognition performance.
2Measurement precision
If word segmentation processing is performed, then the speech recognition accuracy can be improved, but the processing time and computational complexity increase
Solution Approach 1:
The patent extracts the word segmentation function from the traditional processing pipeline and integrates it directly into the neural network model through phoneme-to-character mapping. This extraction eliminates the need for separate post-processing word segmentation steps, thereby maintaining high accuracy while reducing processing time.
Solution Approach 2:
The model performs preliminary character and punctuation prediction based on phoneme inputs during the forward inference process. By conducting these predictions in advance as part of the core modeling process rather than as subsequent processing steps, the patent reduces overall processing time while maintaining accuracy.
3Measurement precision
If a large-sized neural network model is used, then the prediction accuracy can be improved, but the memory footprint increases and deployment on terminal devices becomes difficult
Solution Approach 1:
By combining character and punctuation prediction tasks into a single multitask model, the patent reduces the total number of parameters and memory requirements compared to deploying separate models for each function. This merging approach maintains prediction accuracy through shared computational layers and efficient parameter utilization.
Solution Approach 2:
The universal multitask neural network model achieves efficient resource utilization by serving multiple functions (character recognition and punctuation prediction) with a single architecture. This multi-functionality reduces the overall memory footprint and computational resources needed compared to specialized single-purpose models, enabling deployment on terminal devices with limited memory.
4Measurement precision
If punctuation prediction is added to the speech recognition model, then the prediction accuracy for punctuation can be improved, but the model complexity increases
Solution Approach 1:
The patent merges punctuation prediction with character recognition in a unified multitask neural network. This integration allows the model to learn complementary patterns for both tasks simultaneously, improving punctuation prediction accuracy while avoiding the added complexity of separate punctuation prediction models.
Solution Approach 2:
The neural network model is designed as a universal system that handles both character and punctuation prediction through shared computational layers. This multi-functional design improves punctuation accuracy through joint training while minimizing additional model complexity by reusing existing architectural components for both tasks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This application relates to a speech recognition method and apparatus. The speech recognition method includes: A terminal device inputs a to-be-recognized phoneme into a first multitask neural network model; the first multitask neural network model outputs a first prediction result, where the first prediction result includes a character prediction result and a punctuation prediction result that correspond to the to-be-recognized phoneme; and the terminal device displays at least a part of the first prediction result on a display of the terminal device. A neural network model for simultaneously predicting a character and a punctuation corresponding to a phoneme is constructed, so that the character and the punctuation corresponding to the phoneme can be simultaneously output. In addition, the neural network model is small-sized, and can be deployed on a terminal side.