Shared Feature Extraction for Multi-Dimensional Prosody Prediction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing speech synthesis technologies struggle to accurately predict prosody features, leading to unnatural generated audio, as they fail to effectively incorporate linguistic information and emotional nuances.

Innovation Solution

A prosody prediction model comprising a feature extraction network and multiple feature prediction networks is used to extract and predict prosody features such as pitch stress, phrase stress, and boundary tone, utilizing a pre-trained model like ELECTRA for linguistic information extraction and multi-task learning.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If existing speech synthesis technologies are used, then speech synthesis can be performed, but the generated audio is unnatural due to inaccurate prosody prediction

Engineering Contradiction:

Improveprosody prediction accuracyVSAvoidnaturalness of generated audio

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The prosody prediction model is segmented into multiple independent prediction networks, each dedicated to predicting a specific prosody dimension (pitch stress, phrase stress, boundary tone, break index). This segmentation allows each network to specialize in capturing the unique characteristics of its target dimension, thereby improving overall prediction accuracy and naturalness of synthesized speech.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The feature extraction network serves as a universal component that extracts linguistic information applicable to all prosody dimensions. By sharing this extraction mechanism across multiple prediction networks, the system efficiently leverages common linguistic features for different prosody aspects, improving both accuracy and computational efficiency.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Measurement precision

If a single prosody prediction model is used, then the model structure is simple, but it fails to capture multiple prosody dimensions accurately

Engineering Contradiction:

Improveprosody feature prediction accuracyVSAvoidmodel structure complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The model is divided into a shared feature extraction network and multiple specialized prediction networks. This segmentation enables each prediction network to focus on specific prosody dimensions while sharing common linguistic feature extraction, achieving high prediction accuracy without excessive complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The feature extraction network is merged and shared across all prediction networks. This combining approach allows efficient reuse of linguistic information extraction capabilities, reducing redundant computations and managing model complexity while maintaining accurate multi-dimensional prosody prediction.

Inventive Principle:

Principle #5Merging (Combining)

3Measurement precision

If linguistic information is not effectively incorporated, then the model is simpler to train, but the prosody prediction accuracy decreases

Engineering Contradiction:

Improveprosody prediction accuracyVSAvoidmodel training difficulty

Core Design Contradiction:

Measurement precisionVSEase of manufacture

Solution Approach 1:

Linguistic information is extracted in advance by the feature extraction network before being fed to the prediction networks. This preliminary extraction of linguistic features (such as from text using ELECTRA) prepares the data in a form that is directly useful for prosody prediction, improving accuracy while maintaining manageable training complexity through pre-processing.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250218425A1Prosody prediction method, apparatus, readable medium, and electronic device

Publication Date: 2025.07.03 BEIJING YOUZHUJU NETWORK TECH CO LTD

AI summary

Here provide a method for prosody prediction, an apparatus, a readable medium, and an electronic device. The method includes: obtaining a target text to be processed; and determining prosody feature information of the target text based on the target text and a pre-trained prosody prediction model, the prosody feature information comprising prosody features corresponding to a plurality of predetermined prosody dimensions; wherein the prosody prediction model comprises a feature extraction network and a plurality of feature prediction networks, the feature extraction network being configured to extract linguistic information of the target text, the plurality of feature prediction networks each connected to the feature extraction network and being corresponding to the predetermined prosody dimensions, respectively, and each of the feature prediction networks being configured to predict, based on the linguistic information extracted by the feature extraction network, a prosody feature corresponding to a predetermined prosody dimension.