3D Virtual Human Video Editing With Depth-Aware Face and Pose Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face limitations in synthesizing high-quality, real-time virtual 3D models for live-action videos due to quality deterioration, high production costs, and spatial optimization issues, particularly in editing and reprocessing video content with 3D digital humans.
Innovation Solution
A method for editing performers in videos using a virtual human involves separating a human object from 2D video data, editing it using a 3D virtual human template, and projecting the edited 3D virtual human onto the 2D video, incorporating depth mapping, lighting estimation, and pose synchronization to achieve natural facial expressions and body motions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional 3D digital human technology is used to synthesize virtual models in live-action videos, then the virtual model can be generated, but the production cost increases and the animation quality becomes awkward
Solution Approach 1:
The patent uses a 2D image of the target person as a template to generate the 3D digital human model, rather than creating a full 3D model from scratch. This copying approach significantly reduces production costs while maintaining synthesis quality, as the system leverages existing 2D imagery to create the virtual representation.
Solution Approach 2:
The patent separates the synthesis process into distinct components: face generation from 2D images, body pose estimation, and integration of these elements. This segmentation allows each component to be optimized independently, reducing overall production complexity and cost while improving animation quality.
2Reliability
If full body 3D digital human modeling is performed, then comprehensive virtual representation is achieved, but the costs of modeling, rigging, costumes, and simulations increase
Solution Approach 1:
Instead of creating complete 3D models with full rigging and costumes, the patent copies appearance information from 2D images of the target person and combines it with pose information. This approach achieves comprehensive virtual representation without the high costs associated with traditional full-body 3D modeling pipelines.
Solution Approach 2:
The system uses a universal 2D image template that can be applied to generate 3D representations of different people without requiring separate modeling, rigging, and costume creation for each individual. This multi-functional approach significantly reduces production costs while maintaining representation completeness.
3Manufacturing precision
If 3D face template is selected and 2D face is reflected to generate 3D face, then facial expression quality improves, but processing time increases
Solution Approach 1:
The patent pre-selects and stores multiple 3D face templates before processing. During synthesis, the system quickly matches the target person's 2D face to the most appropriate pre-prepared template, rather than generating a new template each time. This preliminary preparation significantly reduces processing time while maintaining high facial expression accuracy.
4Speed
If real-time face synthesis is implemented, then service delay is reduced, but synthesis quality deteriorates
Solution Approach 1:
The patent segments the synthesis process into independent parallel operations: 2D face analysis, 3D template selection, pose estimation, and model integration. These segmented operations can be executed simultaneously, achieving real-time processing speeds without sacrificing synthesis quality, as each component maintains its optimization.
Data Source
AI summary
A method for editing performers in a video using a virtual human includes: (a) selecting frames including a source human object by searching for a 2D video composed of a plurality of frames; (b) generating a face sequence by tracking a face of the source human object; (c) finding a depth of each object, and determining whether a corresponding object is located in front of or rear of the source human object; (d) selecting a 3D face template, and generating a 3D face of a target human object by reflecting a 2D face of the target human object; (f) estimating pose information, generating a 3D virtual human of the target human object by reflecting the pose information, and synthesizing the 3D face of the target human object with a face portion of the 3D virtual human; and (i) synthesizing the 3D virtual human into a 2D video image space.