AI Voice Recognition with CTC-BERT Cross-Modal Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Collecting large speech-transcription datasets for speech recognition models is time and cost-intensive, limiting the performance of existing speech recognition technologies.
Innovation Solution
A method involving cross-modal learning using a CTC model and a BERT model to improve speech recognition performance by leveraging text embeddings to enhance speech recognition models, utilizing a CTC-BERT scoring method and CMWED loss to train the CTC model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large speech-transcription datasets are collected for training speech recognition models, then model performance is improved, but time and cost investment increases significantly
Solution Approach 1:
The patent introduces text embeddings as an intermediary representation that bridges speech signals and text data. By using text embeddings from pre-trained language models as a mediator, the system can transfer knowledge from text domain to speech domain without requiring extensive speech-transcription pairs, thus resolving the contradiction between model performance and data collection time
Solution Approach 2:
The patent applies pre-training on text data before fine-tuning on speech data. By performing preliminary training on abundant text corpora to establish robust text embeddings, the system prepares the model in advance with general language understanding, reducing the need for extensive speech-specific training data and time
2Reliability
If large speech-transcription datasets are collected for training speech recognition models, then model performance is improved, but monetary cost increases significantly
Solution Approach 1:
The patent uses text embeddings as a computational copy or representation of text data that can be efficiently generated and reused. Instead of collecting and processing expensive speech-transcription pairs, the system creates text embedding representations from readily available text data, reducing monetary costs while maintaining model performance
Solution Approach 2:
The patent makes the speech recognition model capable of utilizing multiple data sources (text corpora and speech data) for training. By designing a unified training framework that accepts both text and speech inputs, the system achieves multi-functionality in data utilization, reducing dependence on expensive speech-transcription datasets
3Quantity of substance
If cross-modal learning with text embeddings is used to improve speech recognition, then data availability is enhanced, but model complexity increases
Solution Approach 1:
The patent merges text processing and speech processing into a unified model architecture. By combining text embedding layers with speech recognition components into a single integrated system, the model can process both modalities simultaneously, enhancing data availability while managing complexity through unified design rather than separate systems
Solution Approach 2:
The patent changes the parameter space by introducing text embeddings as an additional input modality with its own parameter set. By adjusting and optimizing parameters for text processing alongside speech processing parameters, the system expands data utilization capabilities while maintaining controllable model complexity through parameter optimization
Data Source
AI summary
A method performed by an electronic device using artificial intelligence according to an embodiment of the disclosure, the method may include: receiving a first speech signal; and outputting a first text corresponding to the first speech signal from a pre-trained first artificial intelligence algorithm module using the first speech signal as input, wherein the pre-trained first artificial intelligence algorithm module is pre-trained based on a first loss, and wherein the first loss is determined based on a similarity between at least one speech embedding output from the first artificial intelligence algorithm module using a second speech signal as input and at least one text embedding output from a second artificial intelligence algorithm module using a second text as input.