Adaptive Rate Control for Artificial Reality Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality systems face challenges in providing a seamless experience due to latency issues between user movement and image rendering, leading to judder and motion sickness, primarily caused by the inability to efficiently adjust image processing to match display times within strict frame rates.
Innovation Solution
A method and system where a console receives feedback on the completion time of image frames from a head wearable display, adjusts image processing to synchronize completion times with display times, and performs adaptive image processing, such as time warp processing and reprojection, to ensure image frames are generated close to display times, thereby reducing latency and improving user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If image processing is performed at fixed rate by console, then computational resources are conserved and bandwidth efficiency is achieved, but latency between user movement and image rendering increases causing judder and motion sickness
Solution Approach 1:
The patent implements dynamic adjustment of image processing rates based on actual display timing feedback. The system transitions from fixed-rate processing to adaptive processing where the console adjusts its processing rate according to the HWD's actual display completion times, optimizing both resource efficiency and latency performance
Solution Approach 2:
The system establishes a feedback loop where the HWD reports actual display completion times to the console, which then adjusts subsequent image processing rates. This closed-loop control enables the system to respond to actual performance conditions and optimize the balance between resource consumption and rendering latency
2Loss of time
If image processing rate is increased to reduce latency, then user experience is improved, but computational resources and bandwidth consumption increase
Solution Approach 1:
The patent dynamically changes the image processing rate parameter based on feedback from actual display timing. Instead of using a static high processing rate, the system adjusts the processing rate parameter adaptively to match actual display needs, reducing unnecessary computational resource consumption while maintaining low latency when required
3Stability of the object's composition
If strict frame rate timing is enforced, then display synchronization is achieved, but system adaptability to varying processing conditions is reduced
Solution Approach 1:
The system implements dynamic timing adjustment where the console adapts its processing rate based on feedback from the HWD's actual display completion times. This allows the system to maintain display synchronization while being adaptable to varying processing conditions and hardware capabilities
Data Source
AI summary
Disclosed herein are related to systems and methods for providing artificial reality. In one aspect, a console receives feedback information indicative of a first completion time from a head wearable display (HWD). The HWD may complete generating a first image frame at the first completion time. The HWD may display the first image frame. In one aspect, the console compares a display time, at which the first image frame is displayed by the HWD, and the first completion time. In one aspect, the console adjusts, according to the comparison, image processing, to adjust a second completion time, at which the console completes generating a second image frame. In one aspect, the console generates, via the adjusted image processing, the second image frame to provide to the HWD.