Mixed Reality Headset Latency Reduction via Segmented Video Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality systems face significant latency issues in video processing, particularly with occluded displays, which can cause disorientation and nausea due to noticeable delays in camera and display latency, exceeding the 20 ms threshold that affects hand-eye coordination and immersive experience.
Innovation Solution
A head-wearable device comprising a camera, buffer, and processor that performs combined distortion correction, foreground separation, and chromatic aberration correction to reduce latency, achieving processing times under 1 ms for real scene images and under 100 microseconds for virtual scene images, thereby minimizing end-to-end camera latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If separate cameras and traditional video processing methods are used, then real scene images can be captured and displayed, but latency exceeds 20 ms causing disorientation and nausea
Solution Approach 1:
The patent segments the video processing pipeline into distinct functional modules: camera capture, distortion correction, foreground separation, blending, and display output. This segmentation allows each module to be optimized independently for speed, with the buffer strategically positioned to decouple the camera frame rate from the display refresh rate, thereby reducing overall system latency below the 20 ms threshold.
Solution Approach 2:
The patent implements preliminary action by pre-correcting distortion in the real scene images captured by the camera before they enter the main processing pipeline. By performing distortion correction upfront and using a buffer to store pre-processed frames, the system minimizes the processing time required during each display refresh cycle, ensuring latency remains below the perceptible threshold.
2Manufacturing precision
If high-resolution video processing is performed to create immersive mixed reality experience, then blending quality improves, but processing time increases consuming most of the 20 ms latency budget
Solution Approach 1:
The patent performs preliminary distortion correction on high-resolution real scene images before they enter the blending pipeline. This pre-processing step ensures that when the images are later blended with virtual scene images, the operation can proceed efficiently without requiring additional time-consuming distortion corrections during the critical display refresh window, thus maintaining both high blending quality and low latency.
Solution Approach 2:
The patent maintains continuous processing of video frames through an optimized pipeline where distortion correction, foreground separation, and blending operations overlap in time. The buffer allows the system to continuously feed pre-processed real scene images into the blending operation while simultaneously capturing new frames, ensuring that high-resolution processing does not create idle time or bottlenecks that would increase latency.
3Illumination intensity
If camera exposure time is increased to capture sufficient light, then image quality improves, but latency increases as camera and display I/O time consumes most of the 20 ms budget
Solution Approach 1:
The patent segments the I/O operations by decoupling camera frame capture from display frame refresh through the use of a buffer. This allows the camera to operate at its optimal exposure time for image quality without being constrained by the display refresh rate. The buffer accumulates camera frames and releases them at the display's optimal timing, eliminating the need to reduce camera exposure time to meet display timing constraints.
Solution Approach 2:
The patent performs preliminary processing of camera-captured frames, including distortion correction and foreground separation, before the frames need to be displayed. This pre-processing allows the system to use longer camera exposure times for improved image quality while still meeting the display's timing requirements, as the heavy processing is completed in advance rather than during the critical display window.
4Measurement precision
If distortion correction and chromatic aberration correction are performed separately, then correction accuracy improves, but processing latency increases
Solution Approach 1:
The patent merges the distortion correction and chromatic aberration correction operations into a single integrated processing stage. By combining these corrections that were traditionally performed separately, the system achieves the same correction accuracy while reducing the total processing time, as the merged operation eliminates redundant processing steps and allows for more efficient implementation in the video pipeline.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A head wearable device, a method, and a system. The head wearable device includes a display, a camera, a buffer, and a processor. The buffer is configured to buffer a portion of real scene image data corresponding to a real scene image from the camera. The processor is configured to: perform a combined distortion correction operation; perform a foreground separation operation; perform a smoothing operation on blending values; perform a chromatic aberration distortion correction operation; receive virtual scene image data corresponding to a virtual scene image; blend processed real scene image data with the virtual scene image data to create a mixed reality scene image as mixed reality scene image data; and output the mixed reality scene image data to the display for presentation to a user.