Mixed Speech-Text Sequence Training for Limited Paired Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited availability of speech-text pair data hinders effective model training for speech and text generation tasks, impacting the efficiency of generative artificial intelligence models.
Innovation Solution
A method for joint training that involves constructing mixed sequences by aligning text and speech tokens, allowing for the generation of numerous combinations through token replacement and insertion, thereby expanding the dataset for model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If speech-text pair data is used for model training, then the model can generate speech based on text or text based on speech, but the limited availability of such data hinders effective model training
Solution Approach 1:
The patent segments speech and text data into separate token sequences, allowing independent processing and recombination. By dividing the speech sequence and text sequence into discrete tokens that can be independently manipulated, the system creates multiple training sequences from limited paired data, effectively increasing the quantity of training material while maintaining the cross-modal generation capability
Solution Approach 2:
The patent merges speech tokens and text tokens into mixed sequences that combine both modalities. By integrating aligned tokens from speech and text sequences into unified training sequences, the system enables the model to learn cross-modal relationships while maximizing the utilization of limited speech-text pair data through various combination strategies
2Reliability
If more speech-text pair data is collected for training, then model training effectiveness improves, but the cost and complexity of data collection and alignment increases
Solution Approach 1:
The patent implements dynamic sequence construction where the composition of training sequences varies based on alignment relationships. Instead of requiring fixed, pre-aligned speech-text pairs, the system dynamically creates mixed sequences by selectively combining tokens based on their alignment, allowing flexible adaptation to available data while maintaining training effectiveness
Solution Approach 2:
The patent changes the parameter of sequence composition by introducing mixed sequences with varying ratios of speech and text tokens. By adjusting the proportion and arrangement of different token types in training sequences, the system achieves effective cross-modal training without requiring large amounts of perfectly aligned paired data, thereby reducing data collection complexity
Data Source
AI summary
Embodiments in the disclosure relate to joint training. A method provided herein includes: obtaining a first sequence and a second sequence, wherein the first sequence is generated based on text content and the second sequence is generated based on speech content matching the text content, wherein the first sequence includes a plurality of text tokens and the second sequence includes a plurality of speech tokens; constructing a mixed sequence based on an alignment relationship between the plurality of text tokens and the plurality of speech tokens, the mixed sequence including at least one of the plurality of text tokens and at least one of the plurality of speech tokens; and training a target model with the mixed sequence.