Voice Conversion Using Phonetic Posteriorgrams and LSTM Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice conversion methods rely heavily on parallel training data, which is costly to collect, and often suffer from alignment errors due to the need for frame alignment processes like DTW, limiting their effectiveness with non-parallel data.
Innovation Solution
A computer-implemented method using phonetic posteriorgrams (PPGs) generated by a speaker-independent automatic speech recognition (SI-ASR) system, combined with a bidirectional long short-term memory (LSTM) network to model the mapping between PPGs and acoustic features, allowing for voice conversion without the need for parallel data or frame alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If parallel training data is used for voice conversion, then conversion quality is improved, but data collection cost and complexity increase
Solution Approach 1:
The patent introduces phonetic posteriorgrams (PPGs) as an intermediary representation that bridges source and target speech without requiring parallel data. PPGs capture phonetic content in a speaker-independent manner, allowing the conversion model to learn speaker characteristics separately from linguistic content, thereby eliminating the need for costly parallel data collection while maintaining conversion quality
Solution Approach 2:
The patent segments the voice conversion task into two independent components: phonetic content representation (via PPGs) and speaker characteristic mapping. This segmentation allows training on non-parallel data by separately modeling phonetic invariance and speaker variability, resolving the contradiction between conversion quality and data collection complexity
2Measurement precision
If frame alignment processes like DTW are used, then conversion accuracy is improved, but processing time and computational cost increase
Solution Approach 1:
The patent performs preliminary action by generating phonetic posteriorgrams that are inherently aligned to phonetic events rather than requiring post-hoc frame alignment. The PPG representation captures phonetic transitions in a time-ordered sequence that naturally corresponds between source and target speech, eliminating the need for computationally intensive DTW alignment while preserving conversion accuracy
Solution Approach 2:
The patent replaces the mechanical frame alignment process (DTW) with a neural network-based approach that learns temporal correspondence directly from data. The bidirectional LSTM model automatically handles temporal alignment by processing sequences in both forward and backward directions, substituting the iterative mechanical alignment process with a more efficient learned transformation
3Manufacturing precision
If speaker-dependent models are used, then voice conversion quality is improved, but model adaptability to different speakers decreases
Solution Approach 1:
The patent creates a universal phonetic representation (PPGs) that is speaker-independent, allowing the same model to process speech from any speaker. The PPG encoder learns phonetic content that transcends individual speaker characteristics, enabling the conversion system to adapt to new speakers without retraining, thereby achieving both high conversion quality and broad speaker adaptability
Data Source
AI summary
A method for converting speech using phonetic posteriorgrams (PPGs). A target speech is obtained and a PPG is generated based on acoustic features of the target speech. Generating the PPG may include using a speaker-independent automatic speech recognition (SI-ASR) system for equalizing different speakers. The PPG includes a set of values corresponding to a range of times and a range of phonetic classes, the phonetic classes corresponding to senones. A mapping between the PPG and one or more segments of the target speech is generated. A source speech is obtained, and the source speech is converted into a converted speech based on the PPG and the mapping.