Emotion-Editable Digital Human Generation From Speech Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current digital human generation technologies lack the ability to generate digital humans with emotions, limiting their effectiveness in interactive scenarios such as product service introductions, medical rehabilitation, and autonomous driving.
Innovation Solution
A method involving encoding input speech and target emotion information to obtain speech and emotion code matrices, fusing these matrices, and inputting the fusion matrix into a digital human generation model to generate a digital human image with emotions, allowing for emotion editing based on scenario needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current digital human generation methods (GAN or NeRF) are used, then high quality digital humans with aligned mouth shape and voice can be generated, but digital humans with emotions cannot be generated
Solution Approach 1:
The patent segments the emotion generation process into distinct components: emotion type identification (categorizing emotions into discrete classes), emotion intensity measurement (quantifying emotional strength), and separate processing streams for different emotional dimensions. This segmentation allows the system to handle complex emotional expressions through modular, manageable stages, enabling emotional capability without compromising generation reliability
Solution Approach 2:
The patent introduces an emotion code matrix as an intermediary representation that bridges speech input and digital human generation. This intermediate structure encodes both emotion type and intensity information in a structured format that can be seamlessly integrated into existing generation pipelines, allowing emotional expressions to be incorporated without disrupting the proven GAN/NeRF generation processes
2Ease of operation
If digital humans can only express in voice, then generation simplicity is maintained, but user interactive needs cannot be met
Solution Approach 1:
The patent implements dynamic emotional expression by allowing the digital human's emotional state to change in real-time based on speech content and context. The system dynamically adjusts emotion type and intensity parameters during generation, enabling the digital human to adapt its expressions to match the interactive context while maintaining a relatively simple underlying generation framework
Solution Approach 2:
The patent performs preliminary emotion analysis on the input speech to identify emotion type and intensity before the actual digital human generation process. This preliminary action prepares emotion codes in advance, allowing the main generation system to remain simple while incorporating pre-computed emotional information that enhances interactive capability
Data Source
Figure 1
Figure 2~3
AI summary
A digital human generation method and apparatus, a computer-readable storage medium, and a terminal, the method comprising: determining input speech and target emotion information (S11); encoding the input speech to obtain a speech encoding matrix, and encoding the target emotion information to obtain an emotion encoding matrix (S12); fusing the speech encoding matrix and the emotion encoding matrix to obtain a fusion matrix (S13); and inputting the fusion matrix into a digital human generation model to obtain a digital human image corresponding to the input speech (S14). By using the foregoing solution, a digital human with emotion can be generated, and emotion editability of the digital human is achieved.