Deep Learning Speech Synthesis Prosodic Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large model speech synthesis technology faces challenges such as unstable synthesis effects, limitations in data utilization, and difficulties in generating speeches with specific speaker features, due to issues like missing words, repetition, and abnormal prosody.
Innovation Solution
The method involves determining a reference speech feature associated with prosodic features from a sample speech, retrieving a pronunciation expression feature from a speech library using a corresponding text, and adjusting the parameters of a deep learning model based on the difference between the reference and output speech features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the amount of training data is increased, then the data utilization is improved, but the model performance does not improve and computational costs increase
Solution Approach 1:
The patent applies preliminary action by pre-processing and extracting prosodic features from a large corpus of speech data before training the model. The prosodic feature extractor analyzes pitch, stress, and rhythm patterns in advance, creating a structured feature representation that can be directly used for training, thereby avoiding the need to process and train on raw audio data directly.
Solution Approach 2:
The patent extracts the prosodic feature representation from the speech data and separates it from the raw audio signal. By taking out only the essential prosodic information (pitch contours, stress patterns, rhythm) and using this extracted feature for training, the system achieves better model performance with less data, avoiding the computational burden of processing entire audio files.
2Quantity of substance
If synthetic data is used for training, then data utilization is improved, but the data quantity is limited to hundreds or thousands of hours
Solution Approach 1:
The patent changes the parameter representation from raw audio waves to extracted prosodic features. By transforming the data into a feature space that captures essential speech characteristics (pitch, stress, rhythm), the system can effectively utilize smaller amounts of synthetic data, as the feature extraction process concentrates the information density and removes redundant audio data.
3Reliability
If end-to-end speech synthesis model is trained, then speech synthesis effect is improved, but the training reaches an upper limit in data utilization
Solution Approach 1:
The patent segments the speech synthesis task into separate functional components: a prosodic feature extractor that analyzes pitch, stress, and rhythm patterns, and a speech synthesis model that generates audio from these features. This segmentation allows each component to be optimized independently, with the feature extractor handling data processing and the synthesis model focusing on audio generation, thereby improving overall data utilization efficiency.
Data Source
AI summary
A method of training a deep learning model and a method of synthesizing a speech are provided, which relate to a field of artificial intelligence technology, in particular to fields of large model, large language model, generative model, deep learning, and speech processing technologies. The method of training a deep learning model includes: determining a reference speech feature of a sample speech, the reference speech feature being associated with a prosodic feature of the sample speech; retrieving a speech library using a sample text corresponding to the sample speech, so as to obtain a pronunciation expression feature of the sample text; inputting the pronunciation expression feature into the deep learning model to obtain an output speech feature; determining a loss of the deep learning model according to the reference speech feature and the output speech feature; and adjusting a parameter of the deep learning model according to the loss.