Digital Human Action Parameter Adjustment for Natural Transitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital human technologies face challenges in achieving realistic and delicate action changes, particularly when transitioning from a static state to dynamic actions, and face swapping in virtual streamers is inefficient, leading to poor real-time performance.
Innovation Solution
A method for driving digital humans involves obtaining a target action from text, modifying its parameters based on a reference action, and executing the action using an end-to-end model to enhance realism and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a 3D static model is made very realistic through high calculation data, then the realism of the digital human is improved, but the calculation time increases significantly leading to poor real-time performance
Solution Approach 1:
The patent segments the digital human system into a high-quality static 3D model (pre-calculated with high realism) and a real-time driving module (using action parameters from reference actions). This separates the heavy computational task of model construction from the real-time action execution, allowing the static model to be pre-optimized for realism while the dynamic portion operates efficiently in real-time.
Solution Approach 2:
The patent performs preliminary construction of the 3D static model with high realism using extensive calculation data before real-time operation. The model is pre-built and stored, so during real-time operation, the system only needs to drive the pre-built model with action parameters rather than performing complex calculations again, thus achieving both high realism and real-time performance.
2Speed
If the digital human executes a target action abruptly without reference to previous actions, then the response speed is improved, but the naturalness and delicacy of action transition deteriorates
Solution Approach 1:
The patent retrieves a reference action that occurs before the target action in the reference video, and uses this reference action as a baseline for modifying the target action parameters. This preliminary reference action provides a natural transition context, allowing the digital human to execute actions smoothly and naturally while still responding to input in real-time.
Solution Approach 2:
The patent modifies the target action parameters based on the reference action parameters, adjusting parameters such as action intensity, speed, and timing. This parameter modification approach allows the digital human to transition smoothly from the reference action to the target action, maintaining naturalness and delicacy while preserving responsive behavior.
3Adaptability or versatility
If face swapping is performed on historical video material of virtual streamers, then the versatility of virtual streamer applications is improved, but the processing time increases leading to poor real-time performance
Solution Approach 1:
The patent uses a candidate virtual face as a template or copy that is pre-prepared and stored. Instead of generating a new virtual face from scratch for each historical video, the system reuses and adapts the candidate virtual face through parameter modification, significantly reducing processing time while maintaining versatility in different application scenarios.
Solution Approach 2:
The patent modifies parameters of the candidate virtual face to adapt it to different historical video materials and virtual streamer contexts. By changing parameters such as facial expressions, poses, and lighting conditions rather than performing complete face generation, the system achieves versatile application while maintaining fast processing speeds suitable for real-time performance.
Data Source
AI summary
The present disclosure discloses a method and an apparatus for driving a digital human, and an electronic device. The method includes obtaining a target action corresponding to a target text; obtaining a reference action to be executed before the digital human executes the target action when the digital human is driven to output speech based on the target text; modifying a target action parameter of the target action according to a reference action parameter of the reference action; and driving the digital human to execute the target action according to a modified target action parameter when driving the digital human to output the speech based on the target text.