Audio Generation Models for Parallel Speech Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training speech processing models for tasks like speech-to-speech voice conversion requires a large amount of rare and difficult-to-obtain parallel data, limiting their performance and effectiveness.
Innovation Solution
A system generates synthetic audio signals using an audio generation model to create a set of training data, allowing the speech processing model to be trained without extensive parallel data, enabling improved performance and real-time voice conversion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parallel training data is collected manually, then model training quality is improved, but data acquisition time and cost increase significantly
Solution Approach 1:
The patent uses audio generation models to synthesize parallel training data by copying and transforming existing audio content. The system generates synthetic audio signals that mimic the characteristics of target speakers without requiring manual collection of actual speech samples from each speaker, thus maintaining training quality while dramatically reducing data acquisition time
Solution Approach 2:
The system employs self-supervised learning where the audio generation model automatically generates its own training data without external human intervention. The model uses its internal representations and speaker embeddings to create synthetic parallel data, eliminating the need for manual data collection and annotation processes
2Reliability
If more parallel training data is obtained, then speech processing model performance is improved, but data availability decreases due to rarity and difficulty of acquisition
Solution Approach 1:
The patent transforms the availability parameter by changing from physical data collection to synthetic data generation. By adjusting parameters such as speaker embeddings, audio quality settings, and synthesis conditions, the system can generate unlimited parallel training data with varying characteristics, making the data effectively unlimited and highly adaptable
Solution Approach 2:
The audio generation model serves multiple functions: it acts as both the processing engine and the data generation engine. The same model architecture used for speech processing is leveraged to generate training data, eliminating the need for separate data collection systems and enabling the system to adapt to different speakers and conditions using a single universal framework
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a set of training data for training a speech processing model. One of the methods may include receiving a plurality of source audio signals that each represent speech; generating, for each source audio signal, a respective semantic representation of the source audio signal; obtaining, for each of a plurality of speakers, a respective speaker prompt embedding characterizing speech of the speaker; generating, for each source audio signal, one or more synthetic audio signals; and generating a set of training data for training a speech processing model, wherein the set of training data comprises a plurality of paired training examples.