Latency Compensation via Pre-captured Image Buffers in AR Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial reality systems face image-processing latency issues when rendering a user's body movements in real-time, leading to delayed visual feedback, which is perceptible and disrupts the immersive experience.
Innovation Solution
The system employs a combination of cameras and sensors to capture and reconstruct the external environment in 3D, using deformation vectors to synchronize the virtual environment with real-time controller positions, thereby eliminating perceived latency by adjusting the virtual environment's geometry based on current controller orientation and position.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system uses cameras to capture and render the external environment in real-time, then the virtual environment can be updated continuously, but image-processing latency causes delayed visual feedback
Solution Approach 1:
The system performs preliminary actions by capturing images at multiple past time points (t-1, t-2, t-3) in advance and storing them for later use. This allows the system to compensate for processing latency by having pre-captured images ready to be displayed at the current time point, effectively eliminating the perceptible delay in visual feedback.
Solution Approach 2:
The system maintains continuous image capture and rendering operations by continuously updating the virtual environment with images from multiple past time points. This continuous process ensures that the display always shows the most recent available visual information, maintaining immersion while accounting for processing delays.
2Measurement precision
If the system renders the external environment using captured images, then the virtual environment reflects the real world, but latency makes it difficult to capture user body movements in real-time
Solution Approach 1:
The system segments the rendering process into multiple components: capturing images at different time points, storing them in a buffer, and selectively displaying them based on latency compensation. This segmentation allows the system to maintain high visual accuracy while independently managing the timing of movement responses.
Solution Approach 2:
The system implements feedback by continuously monitoring user body movements through the virtual environment and adjusting the display accordingly. The latency compensation mechanism ensures that the feedback loop is closed by using pre-captured images that correspond to the user's movements at the actual time they occurred, not at the delayed rendering time.
3Adaptability or versatility
If the system uses a triangular mesh to represent the virtual environment, then the rendering can be stylized or photorealistic, but the mesh must be deformed to compensate for latency
Solution Approach 1:
The system performs preliminary deformation calculations by determining deformation vectors in advance based on the positions and orientations of controllers at past time points. These pre-calculated deformations are then applied to the triangular mesh to compensate for latency, allowing the system to maintain rendering flexibility while managing computational complexity through advance preparation.
Data Source
AI summary
In one embodiment, a method includes receiving image data corresponding to an external environment of a user. The image data is captured at a first time and comprises a body part of the user. The method also includes receiving a first tracking data generated based on measurements made at the first time by at least one motion sensor associated with the body part; generating, based at least on the image data, a model representation associated with the body part; receiving a second tracking data generated based on measurements made at a second time by the at least one motion sensor associated with the body part; determining a deformation of the model representation associated with the body part based on the first tracking data and the second tracking data; and displaying the deformation of the model representation associated with the body part of the user.