Speech Recognition Pre-Training Using Permutation Invariant Cross-Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end speech recognition models require large amounts of transcription data for training, which is costly and labor-intensive, and existing self-supervised learning methods like wav2vec and vq-wav2vec do not effectively align pre-training and fine-tuning steps.
Innovation Solution
A self-supervised learning method using permutation invariant cross entropy (PICE) trains on non-transcription speech data by defining a cross entropy loss function and calculating minimum losses for all permutations of classes, aligning pre-training and fine-tuning steps without transcription data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If supervised learning is used to train end-to-end speech recognition models, then good recognition performance is achieved, but large amounts of transcription data are required which increases cost and effort
Solution Approach 1:
The patent applies preliminary action by performing self-supervised pre-training on speech representations using only audio data before fine-tuning with transcription data. This preliminary pre-training step extracts useful speech features without requiring transcription labels, reducing the amount of annotated data needed for subsequent supervised training while maintaining recognition performance.
Solution Approach 2:
The patent implements self-service through self-supervised learning where the model learns from the speech data itself without external transcription labels. The system uses permutation invariant cross-entropy loss to create learning signals from unlabelled speech corpus, enabling the model to serve its own training needs without requiring costly manual transcription.
2Quantity of substance
If existing self-supervised learning methods like wav2vec are used, then transcription data requirement is reduced, but pre-training and fine-tuning steps do not align effectively
Solution Approach 1:
The patent applies parameter changes by modifying the loss function parameters to be permutation invariant. The cross-entropy loss is reformulated to remain unchanged under permutation of class labels, creating a training objective that is inherently aligned between pre-training and fine-tuning stages. This parameter modification ensures consistent learning behavior across both training phases without requiring complex alignment mechanisms.
3Quantity of substance
If non-transcription speech corpus data is used for pre-training, then training cost is reduced, but the model needs to learn inherent expression without output targets
Solution Approach 1:
The patent introduces an intermediary mechanism by using permutation invariant cross-entropy loss as a mediator between unlabelled speech data and model training. This loss function acts as an intermediary that creates learning signals from unlabelled data by comparing predicted distributions across permutations, enabling training without direct output targets while maintaining computational tractability.
Data Source
AI summary
Provided is a self-supervised learning method based on permutation invariant cross entropy. A self-supervised learning method based on permutation invariant cross entropy performed by an electronic device includes: defining a cross entropy loss function for pre-training of an end-to-end speech recognition model; configuring non-transcription speech corpus data composed only of speech as input data of the cross entropy loss function; setting all permutations of classes included in the non-transcription speech corpus data as an output target and calculating cross entropy losses for each class; and determining a minimum cross entropy loss among the calculated cross entropy losses for each class as a final loss.