Inter-Eye Prediction Models for Low-Latency XR Video Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR devices face challenges with high video encoding latency and processing overhead, particularly in devices with multiple cameras, where transform functions for correlating camera perspectives are lacking, leading to inefficient resource utilization.
Innovation Solution
Implementing inter-eye prediction models for joint source channel encoding schemes that utilize geometry and gaze direction to enhance compression and reduce power consumption in XR devices, specifically for devices with multiple cameras aligned with user eyes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional video encoding is used in XR devices with multiple cameras, then video quality can be maintained, but encoding latency and processing overhead increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-establishing prediction models during device calibration that capture the geometric relationships and gaze direction correlations between multiple cameras. These pre-computed models enable the encoding system to predict inter-eye video content without real-time complex processing, thus maintaining video quality while reducing encoding latency during actual XR operation
Solution Approach 2:
The patent uses copying by creating prediction models that replicate the correlation patterns between camera perspectives based on gaze direction. Instead of processing all camera data equally during encoding, the system copies relevant predictive information from calibrated models to reconstruct video content, significantly reducing the computational burden during real-time encoding while preserving visual fidelity
2Productivity
If complex transform functions are implemented to correlate multiple camera perspectives, then video encoding efficiency improves, but device power consumption increases
Solution Approach 1:
The patent reduces power consumption during video encoding by performing complex transform function calculations in advance during device calibration. The prediction models capturing camera perspective correlations are pre-computed and stored, allowing the encoding process to use these pre-prepared models rather than performing intensive calculations in real-time, thus maintaining encoding efficiency while minimizing energy usage during XR operation
Solution Approach 2:
The patent implements self-service by having the system calibrate and create its own prediction models during initial setup or idle periods when power availability is less critical. These self-generated models then serve the encoding process autonomously, eliminating the need for continuous high-power processing during video capture and encoding operations
3Device complexity
If inter-eye prediction models are implemented, then processing overhead is reduced, but device complexity increases due to additional calibration requirements
Solution Approach 1:
The patent reduces calibration complexity by implementing self-calibration mechanisms where the system automatically determines camera geometric relationships and gaze direction correlations through automated procedures during initial setup. This self-service approach eliminates or minimizes the need for manual calibration processes, making the additional complexity transparent to users while still enabling the prediction models to be created accurately
Data Source
AI summary
Method and apparatus for inter-eye prediction models for XR. The apparatus transforms at least part of a first frame associated with a first perspective of a first camera to at least part of a second frame associated with a second perspective of a second camera based on at least one prediction model. The apparatus transmits, to a second wireless device, a prediction indication of the at least one prediction model used to transform the at least part of the first frame to the at least part of the second frame. The apparatus transmits, to the second wireless device, an encoded signal comprising a combined frame comprising a combination of at least the first frame and the second frame.