Speech Model Training Using Pseudo Tokens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training speech processing models requires large amounts of labeled data, which is time-consuming and expensive to obtain, limiting the efficiency and cost-effectiveness of the process.
Innovation Solution
The method involves using unlabeled data to compute pseudo tokens through clustering and token compression, pre-training a neural network with these pseudo tokens, and then fine-tuning it with labeled data to reduce the need for extensive labeled training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of labeled training data are used to train speech processing models, then the quality and accuracy of the model is improved, but the cost and time required to obtain the labeled data increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-training the speech processing model on unlabeled audio data before fine-tuning with labeled data. The pre-training phase extracts meaningful features and learns speech patterns from abundant unlabeled data, preparing the model in advance so that subsequent training with limited labeled data can achieve high quality more efficiently.
Solution Approach 2:
The patent uses pseudo-language as an intermediary between raw audio data and final text output. During pre-training, the model learns to map audio to pseudo-language tokens, creating an intermediate representation that captures speech characteristics. This intermediary layer enables more efficient learning from labeled data in the fine-tuning phase.
2Measurement precision
If large amounts of labeled training data are used to train speech processing models, then the accuracy of the model is improved, but the cost to obtain the labeled data increases
Solution Approach 1:
The model performs preliminary learning on unlabeled data to acquire general speech patterns and acoustic features before being exposed to labeled data. This preliminary action enables the model to reach high accuracy with a smaller volume of expensive labeled training data.
Solution Approach 2:
The system uses unlabeled data to self-supervise the model during pre-training, allowing the model to learn useful representations without human annotation. This self-service approach reduces dependency on large volumes of labeled data while maintaining model accuracy.
3Productivity
If traditional training methods using only labeled data are used, then the model learns accurate speech-to-text mapping, but the training efficiency and cost-effectiveness deteriorates
Solution Approach 1:
The two-stage training process applies preliminary action by conducting pre-training on unlabeled data before fine-tuning with labeled data. This preliminary phase accelerates convergence and improves training efficiency, while the subsequent fine-tuning ensures high model performance on actual speech-to-text tasks.
Solution Approach 2:
The patent merges two training approaches: pre-training on unlabeled data and fine-tuning on labeled data. This combination leverages the benefits of both approaches - the efficiency and scalability of using unlabeled data, and the accuracy and task-specific performance of labeled data training - achieving both high productivity and reliability.
Data Source
AI summary
A speech processing model may be trained using pseudo tokens. Training a speech processing model with pseudo tokens may allow for training with a smaller amount of labeled training data and accordingly lower costs. A set of pseudo tokens may be determined by computing feature vectors from unlabeled training data, clustering the feature vectors, and performing token compression using the clustered feature vectors. A first speech processing model may be trained using unlabeled training data by determining sequences of pseudo tokens corresponding to the unlabeled training data. A second speech processing model may be initialized using the first speech processing model and then trained using labeled training data. The second speech processing model may then be deployed to a speech processing application.