Hybrid Visual Communication Using 3D Mesh Model Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video communication systems require high bandwidth and latency, fail to convey 3D depth information, and often result in users appearing to look down due to camera/display geometry disparities, while 2D image sequences do not accurately represent a user's likeness or motion.
Innovation Solution
A hybrid visual communication system that creates a 3D mesh model of a user, updates it using sensor data, and transmits these updates, allowing for real-time display of facial expressions and motion with reduced bandwidth requirements, enabling users to appear as if looking each other in the eye even in bandwidth-constrained environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 2D video streams are transmitted between devices, then video communication capability is provided, but bandwidth consumption is high and latency is high
Solution Approach 1:
The patent segments the video transmission into two parts: a base layer containing essential 3D mesh model data for structural representation, and enhancement layers containing additional visual details. This segmentation allows receivers to reconstruct acceptable video quality using only the base layer, significantly reducing bandwidth consumption while maintaining communication capability.
Solution Approach 2:
Instead of transmitting full 2D video frames, the patent creates a simplified 3D mesh model copy that represents the essential structure and motion of the video content. This mesh model copy requires far fewer bits to transmit while still enabling reconstruction of the original video appearance, thus reducing bandwidth consumption.
2Reliability
If 2D video streams are transmitted between devices, then video communication capability is provided, but latency is high
Solution Approach 1:
The patent performs preliminary actions by pre-processing video content into 3D mesh model representations before transmission. The base layer mesh model is prepared in advance and can be quickly decoded and displayed, reducing the time required for real-time processing and transmission, thus lowering latency.
Solution Approach 2:
The use of simplified 3D mesh model copies instead of full video frames reduces the computational burden of encoding, transmitting, and decoding operations. This copying approach enables faster processing speeds and lower latency while maintaining acceptable video communication quality.
3Quantity of substance
If 3D mesh model updates are transmitted instead of image sequences, then bandwidth is reduced, but system complexity increases
Solution Approach 1:
The patent segments the 3D mesh model data into hierarchical layers: a base layer containing essential structural information and enhancement layers containing additional detail. This segmentation reduces bandwidth by allowing receivers to function with only the base layer, while managing complexity through organized data structures that can be processed incrementally.
Solution Approach 2:
The patent changes the representation parameters from traditional 2D pixel-based video to 3D mesh model parameters (vertices, faces, transformations). This parameter transformation enables more efficient bandwidth utilization while the complexity is managed through standardized mesh formats and progressive refinement techniques.
Data Source
AI summary
Exemplary embodiments for visual communication between a first device and a second device, comprising: creating a 3D mesh model of a first device user; receiving sensor data from a sensor array during the visual communication session between the first device and the second device, wherein the image data includes motion of the first device user; determining 3D mesh model updates using the image data; transmitting the 3D mesh model updates to the second device for the second device to update display of the 3D mesh model of the first device user, wherein the update is represented as one or more of a blend shape and a relative vertex position change of the 3D mesh model.