Synthetic Speech Training Data Using Semantic Audio Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training speech processing models for tasks like speech-to-speech voice conversion requires a large amount of rare and difficult-to-obtain parallel data, limiting their performance and effectiveness.
Innovation Solution
A system generates synthetic audio signals using an audio generation model to create a set of training data, allowing the speech processing model to be trained without extensive parallel data, enabling improved performance and real-time voice conversion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parallel training data is used to train speech processing models, then model performance is improved, but data availability deteriorates due to rarity and difficulty of obtaining parallel data
Solution Approach 1:
The patent creates synthetic parallel training data by copying and transforming existing audio content. Specifically, it generates synthetic audio signals that mimic the characteristics of target speakers while preserving the semantic content of source audio, thereby creating artificial parallel datasets without requiring actual parallel recordings from multiple speakers.
Solution Approach 2:
The patent introduces semantic representations as an intermediary between source audio and synthetic output. This intermediary layer enables the system to separate content preservation from speaker characteristic transformation, allowing generation of realistic synthetic parallel data through conditional generation models that use semantic tokens to guide the synthesis process.
2Reliability
If more training examples are generated to improve model performance, then training data quantity increases, but processing time and computational resources worsen
Solution Approach 1:
The patent performs preliminary processing by generating semantic representations and speaker embeddings before the main synthesis process. This pre-computation of semantic tokens and speaker characteristics allows the conditional generation model to efficiently generate multiple synthetic examples without repeating expensive computation, thereby reducing overall processing time while maintaining data quality.
Solution Approach 2:
The patent segments the training data generation process into distinct components: semantic representation extraction, speaker embedding generation, and conditional audio synthesis. This segmentation allows parallel processing of different audio samples through the same pipeline, improving computational efficiency and reducing total processing time while generating large volumes of training data.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a set of training data for training a speech processing model. One of the methods may include receiving a plurality of source audio signals that each represent speech; generating, for each source audio signal, a respective semantic representation of the source audio signal; obtaining, for each of a plurality of speakers, a respective speaker prompt embedding characterizing speech of the speaker; generating, for each source audio signal, one or more synthetic audio signals; and generating a set of training data for training a speech processing model, wherein the set of training data comprises a plurality of paired training examples.