Pitch-Based Speech Conversion Model Training Method
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge is to improve the pitch similarity between converted speech and real person speech when only a small amount of sample data is available for training the speech conversion model.
Innovation Solution
A pitch-based speech conversion model training method is proposed, which includes a priori encoder, a posteriori encoder, a temporal alignment module, a decoder, and a pitch extraction module. The method involves inputting reference speech to the priori encoder and pitch extraction module to extract audio and pitch features, concatenating these features, aligning them with an audio latent variable, and decoding to obtain converted speech, while iteratively training the model to minimize training loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-training and model fine-tuning are used with a small amount of sample data, then the speech conversion model can be trained, but the pitch similarity between converted speech and real person speech remains low
Solution Approach 1:
The patent extracts pitch features separately from the speech signal using a dedicated pitch extraction module, and then integrates these extracted pitch features into the speech conversion model. This separation allows the model to explicitly learn pitch characteristics without requiring large amounts of sample data, thereby improving pitch similarity in low-data scenarios.
Solution Approach 2:
The patent introduces pitch features as an additional parameter in the speech conversion model, changing the model's parameter structure from only speech features to include both speech features and pitch features. This parameter enhancement enables the model to better control pitch characteristics in converted speech, improving pitch similarity even with limited training data.
2Manufacturing precision
If a large amount of sample data is used to train the speech conversion model, then high-quality speech conversion can be achieved, but it is difficult to obtain sufficient sample data in practice
Solution Approach 1:
The patent performs pre-training on a large-scale speech dataset before fine-tuning on the target speaker's data. This preliminary action allows the model to learn general speech conversion capabilities from abundant data, and then adapt to specific speakers with minimal data, achieving high conversion quality without requiring large amounts of target speaker samples.
Solution Approach 2:
The patent introduces a pitch extraction module as an intermediary component that processes the speech signal to extract pitch features, which are then fed into the speech conversion model. This intermediary mechanism enables the model to focus on learning pitch characteristics separately, improving speech conversion quality while reducing the amount of sample data needed.
3Ease of manufacture
If the speech conversion model is trained with limited data, then training can be completed, but the converted speech has significant gaps from real person speech in timbre and rhythm
Solution Approach 1:
The patent segments the speech signal processing into multiple independent modules: speech feature extraction, pitch feature extraction, temporal alignment, and speech synthesis. This segmentation allows each module to be optimized independently, with the pitch extraction module specifically addressing pitch accuracy even with limited training data, thereby improving overall speech authenticity while maintaining training feasibility.
Solution Approach 2:
The patent introduces a temporal alignment module as an intermediary between pitch feature extraction and speech synthesis. This module aligns the temporal characteristics of pitch features with speech features, ensuring that rhythm and timing are properly synchronized in the converted speech, thereby improving speech authenticity without requiring additional training data.
Data Source
AI summary
The present disclosure provides a pitch-based speech conversion model training method and a speech conversion system, wherein an audio feature code is output by a priori encoder, and a pitch feature is extracted by a pitch extraction module. A linear spectrum corresponding to the reference speech is input into the posteriori encoder to obtain an audio latent variable. In addition, the audio feature code, a speech concatenation feature obtained by concatenation of the audio feature code and the pitch feature, and the audio latent variable are input into a temporal alignment module to obtain a converted speech code, and the converted speech code is decoded by a decoder to obtain a converted speech. The training loss of the converted speech is then calculated to determine the degree of convergence of the speech conversion model.