Virtual Representation Calls With Mesh Streaming for Low-Latency XR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems face challenges in efficiently generating and animating high-quality virtual representations of users in virtual environments, particularly in mobile and wearable devices, due to high computational complexity and resource demands, which affect latency and user experience.
Innovation Solution
A distributed generation system for virtual content that includes an animation and scene rendering system processing input information from multiple devices to generate and animate virtual representations, using efficient communication frameworks for mesh information and animation parameters transmission between devices, reducing data transmission rates and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-quality virtual representations are generated and animated in real-time, then realism and visual quality are improved, but computational complexity and resource demands increase
Solution Approach 1:
The system segments the virtual representation generation into separate components: mesh data generation from input images, texture mapping, and animation parameter extraction. Each component can be processed independently and in parallel, reducing the computational burden on any single device while maintaining high visual quality
Solution Approach 2:
The system performs preliminary actions by pre-generating mesh data and texture maps from user input images before the virtual session begins. This allows complex computational tasks to be completed in advance, so that during the actual virtual session, only lighter animation and rendering operations are required, reducing real-time computational complexity
2Manufacturing precision
If photo-realistic virtual representations are generated, then visual quality is improved, but power consumption increases
Solution Approach 1:
The system extracts only the essential visual features from input images to create simplified mesh representations and texture maps, rather than processing complete high-resolution images continuously. This extraction approach maintains photo-realistic quality for key features while significantly reducing the computational power required for rendering
Solution Approach 2:
The system changes parameters by using compressed mesh data structures and adjustable detail levels for different viewing conditions. It dynamically adjusts the level of geometric and textural detail based on distance, lighting conditions, and device capabilities, maintaining photo-realistic appearance where needed while reducing power consumption in less critical areas
3Loss of time
If virtual representations are generated in real-time with low latency, then responsiveness is improved, but computational resources required increase
Solution Approach 1:
The system merges multiple computational tasks into unified processing pipelines: combining mesh generation, texture mapping, and animation parameter extraction into integrated workflows that share common data structures and processing resources. This eliminates redundant computations and reduces overall processing time and resource requirements
Solution Approach 2:
The system performs preliminary generation of mesh data and base representations before the virtual session starts, so that during the session, only animation updates and rendering are required. This shifts heavy computational work to pre-processing stages, enabling low-latency real-time performance with reduced instantaneous computational resource demands
Data Source
AI summary
Systems and techniques are described for establishing one or more virtual sessions between users. For instance, a first device can transmit, to a second device, a call establishment request for a virtual representation call for a virtual session and can receive, from the second device, a call acceptance indicating acceptance of the call establishment request. The first device can transmit, to the second device, first mesh information for a first virtual representation of a first user of the first device and first mesh animation parameters for the first virtual representation. The first device can receive, from the second device, second mesh information for a second virtual representation of a second user of the second device and second mesh animation parameters for the second virtual representation. The first device can generate, based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user.