Speech Encoder Training Using Text Mask Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech encoders trained through masked pre-training methods produce representations with relatively low semantic levels, leading to low prediction accuracy on downstream tasks requiring high semantic levels.
Innovation Solution
Train the speech encoder by combining speech and text data with a semantic matching relationship, encoding speech data to obtain a speech feature representation, masking text features, predicting masked features using speech data, and adjusting the encoder based on prediction differences to enhance semantic learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If masked pre-training method is used on speech data only, then the speech encoder can be trained efficiently, but the learned representation has low semantic level and poor prediction accuracy on high semantic level tasks
Solution Approach 1:
The patent combines speech data and text data into a unified training framework. The speech encoder and text encoder are trained jointly using multi-modal contrastive learning, where speech representations and text representations are aligned in the same feature space. This merging of modalities allows the speech encoder to learn high-level semantic representations by leveraging the rich semantic information from text data, thereby improving prediction accuracy on high semantic level tasks while preventing semantic information loss.
2Adaptability or versatility
If speech data only is used for training, then the training process is simple, but the representation learning capacity is limited and cannot capture high-level semantics
Solution Approach 1:
The patent creates a multi-functional training framework where a single unified model can process both speech and text data. The speech encoder and text encoder share common components and are trained using a unified loss function that combines contrastive learning objectives. This universal framework expands the representation learning capacity to capture high-level semantics from both modalities while managing complexity through shared architecture and integrated training procedures.
Solution Approach 2:
The patent introduces a contrastive learning mechanism as an intermediary that bridges speech and text modalities. The contrastive loss function acts as a mediator that aligns speech representations with text representations in the feature space, enabling the speech encoder to learn high-level semantics without requiring complex end-to-end multi-modal architecture. This intermediary approach simplifies the overall system complexity while achieving enhanced representation learning.
Data Source
AI summary
Disclosed are a speech encoder training method performed by a computer device. The method includes: masking a first sub-feature representation at a first feature position in a first text feature representation to obtain a first masked feature representation; performing feature prediction on a masked first feature position in the first masked feature representation based on a first speech feature representation to obtain a first predicted feature representation; and training a first speech encoder based on a difference between the first predicted feature representation and the first sub-feature representation to obtain a second speech encoder. The first speech encoder is trained by combining data in a speech modality with data in a text modality, and information included in the data in the text modality is adopted so that the first speech encoder can learn relatively high-level semantic representations of speech, thereby improving the prediction accuracy of representations.