Speech Generation With Feature Fusion for Low-Data Voice Cloning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing TTS and VC models require large amounts of recorded speech data for effective performance, and unsupervised speaker adaptation methods fail to improve performance with increasing data, leading to complex training methods and inadequate task handling.
Innovation Solution
A method involving a speech generation model that acquires a speech feature vector, converts it to a language unit vector, combines with a text feature vector, and uses a sequence-to-sequence model and vocoder to generate target speech, facilitating multimodal modeling and improving performance with a small amount of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of recorded speech data are used for TTS and VC models, then model performance is improved, but training complexity and data requirements increase
Solution Approach 1:
The model is divided into multiple specialized components: acoustic model, vocoder, speaker encoder, and text encoder. Each component processes specific aspects of speech generation independently, allowing the system to achieve high performance with less data while reducing overall training complexity through modular design
Solution Approach 2:
The speech generation model is designed to handle multiple tasks simultaneously - text-to-speech conversion and voice conversion - using a unified architecture that processes different input types (text features and speech features) through the same acoustic model and vocoder, improving efficiency and reducing redundant training
2Quantity of substance
If unsupervised speaker adaptation methods are used, then data requirements are reduced, but performance improvement is insufficient
Solution Approach 1:
The method combines supervised and unsupervised learning approaches by integrating speaker encoding mechanisms that can adapt to speaker characteristics without requiring extensive labeled data. The speaker encoder processes speech features to extract speaker-specific information that enhances performance while maintaining low data requirements
Solution Approach 2:
The system dynamically adjusts model parameters based on input speech features, allowing the acoustic model and vocoder to adapt their behavior to different speakers and conditions. This parameter adaptation enables performance improvement without requiring large amounts of retraining data
3Adaptability or versatility
If complex training methods are employed, then task handling capability is improved, but training duration and complexity increase
Solution Approach 1:
The model incorporates pre-trained components and feature extraction mechanisms that are prepared in advance. The speaker encoder and text encoder are designed with pre-established processing pipelines that reduce the need for extensive training while maintaining high task handling capability
Solution Approach 2:
The training process is made dynamic and adaptive, allowing the model to focus computational resources on the most critical learning tasks. The system automatically adjusts training priorities based on input data characteristics, reducing overall training duration while maintaining versatility
Data Source
AI summary
A method for generating a speech includes acquiring a voice feature vector of a voice to be processed, and inputting the voice feature vector into a voice generation model to obtain a language unit vector; acquiring a text feature vector, and determining, according to the text feature vector and the language unit vector, a feature vector to be processed; and inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target voice corresponding to the voice to be processed or the text feature vector.