Virtual Livestreaming Motion Capture via Real-Time Expression Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual livestreaming methods using inertial capture devices result in poor display quality and are only suitable for low-quality applications due to inaccuracies and limitations in capturing skeletal motions and facial expressions, leading to unnatural transitions and restricted motion capabilities.
Innovation Solution
A method that acquires real feature data of a subject, including motion and face data, to determine target feature data for a virtual character, which is then used to generate video stream pushing data for a livestreaming platform, allowing for more realistic and vivid skeletal motions and facial emotions by avoiding manual animation and incorporating preset motion and special effect data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If inertial capture devices are used to acquire limb motion of a real streamer, then the virtual livestreaming can be generated, but the display effect becomes poor and the quality is limited
Solution Approach 1:
The system segments the motion capture process into multiple independent tracking components (limb motion tracking, face motion tracking, eye expression tracking) rather than relying on a single inertial capture device. This segmentation allows each component to be optimized independently, improving overall measurement precision while maintaining productivity.
Solution Approach 2:
The patent introduces intermediate processing steps including motion data filtering, face data alignment, and expression mapping as mediators between the raw capture data and the final virtual character animation. These intermediaries refine the data quality and improve display effects without compromising the generation capability.
2Ease of operation
If premade expression control is used to drive virtual character, then the livestreaming can be performed, but the transitions become unnatural and motion capabilities are restricted
Solution Approach 1:
The system transitions from static premade expressions to dynamic real-time expression synthesis. The virtual character's facial expressions are dynamically generated based on real-time tracking of the streamer's face motions and eye expressions, creating natural transitions while maintaining ease of operation through automated control.
Solution Approach 2:
The system implements feedback mechanisms where the tracked face data and eye expression data continuously inform and adjust the virtual character's expressions in real-time. This closed-loop feedback ensures natural motion transitions and unrestricted motion capabilities while keeping the operation simple.
3Measurement precision
If optical capture devices are used to track face motions and eye expressions, then the measurement precision improves, but the device complexity increases
Solution Approach 1:
The system uses a camera as a multi-functional device that performs both limb motion tracking and face motion tracking, and also captures eye expressions. This universal approach improves measurement precision across multiple parameters while avoiding the complexity of deploying separate specialized capture devices for each function.
Solution Approach 2:
The patent merges multiple capture functions into a unified processing pipeline where face data, eye expression data, and limb motion data are captured and processed together. This consolidation reduces device complexity by eliminating redundant hardware while maintaining high measurement precision through integrated processing.
Data Source
AI summary
Provided are a virtual livestreaming method, apparatus, and system, and a storage medium, relating to the technical field of livestreaming. The method is acquiring real feature data of a real subject, where the real feature data include motion data and face data of the real subject during a performance; determining target feature data of a virtual character according to the real feature data, where the virtual character is a preset animation model, and the target feature data include motion data and face data of the virtual character; determining video stream pushing data corresponding to the virtual character according to the target feature data; and sending the video stream pushing data corresponding to the virtual character to a target livestreaming platform, where the video stream pushing data is used for instructing the target livestreaming platform to display virtual livestreaming of the virtual character.