3D Object Mesh Prediction for Low-Latency XR Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality (XR) systems face challenges with high power consumption and latency issues due to the split rendering between companion devices and head-mounted displays (HMDs), leading to inconsistent immersive experiences.
Innovation Solution
Implement motion disentanglement techniques for predicting object masks and meshes by generating 3D keypoints, predicting future object locations and poses, and deforming mesh models to reduce latency through multi-sampling poses during reprojection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If split rendering is used between companion device and HMD, then power consumption and device weight are reduced, but latency increases and object consistency deteriorates
Solution Approach 1:
The system performs preliminary actions by predicting future camera poses and object states before they occur. Multiple future poses are predicted in advance, and the appropriate pose is selected during reprojection to compensate for rendering latency, ensuring objects appear at correct positions without requiring real-time processing power.
Solution Approach 2:
The system dynamically adapts to changing conditions by using motion disentanglement that separates camera motion from object motion. This allows the system to adjust object positions based on predicted camera trajectories while maintaining object consistency, resolving the latency issue without requiring constant high-power processing.
2Weight of moving object
If split rendering is used between companion device and HMD, then device weight is reduced, but latency increases and user experience deteriorates
Solution Approach 1:
Multiple future camera poses are predicted in advance using motion disentanglement techniques. During reprojection, the system selects from these pre-predicted poses to compensate for rendering delays, ensuring low-latency object placement without requiring the HMD to perform heavy computational tasks.
Solution Approach 2:
The system creates simplified representations (masks and meshes) of objects that can be efficiently rendered and positioned. These copied object representations are then placed in the virtual environment based on predicted poses, reducing the computational burden on the HMD while maintaining visual fidelity.
3Stability of the object's composition
If motion prediction is performed, then object consistency is maintained, but computational complexity increases
Solution Approach 1:
The system segments motion into separate components: camera motion (derived from pose information) and object motion (predicted independently). This segmentation allows each component to be processed separately using appropriate algorithms, reducing overall computational complexity while maintaining accurate object consistency.
Solution Approach 2:
The system changes parameters by predicting multiple future poses at different time steps and selecting the appropriate pose based on actual camera movement. This parameter-based approach simplifies the computational problem from continuous motion tracking to discrete pose selection, reducing complexity while maintaining accuracy.
Data Source
AI summary
Systems and techniques are provided for object segmentation. For instances, a process can include: obtaining camera pose information for frames of a set of input frames; predicting a future camera pose based on the camera pose information; generating 3-dimensional (3D) keypoints for an object present in the set of input frames; predicting a future location point of the object based on location points associated with the 3D keypoints for the object; predicting a future pose of the object based on the 3D keypoints; deforming a mesh model of the object based on the future pose of the object; and rendering a segmentation image based on the deformed mesh model, the future location point, and the future camera pose.