ER Frame Extrapolation Using Device Motion to Cut Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing enhanced reality (ER) systems face challenges in presenting ER settings at higher frame rates due to the limitations of interpolation, which introduces high latency, making it unsuitable for real-time applications.
Innovation Solution
Implementing frame rate extrapolation techniques that utilize motion data from devices like inertial measurement units and gyroscopes to generate extrapolated frames based on device movement, reducing latency and enabling higher frame rates without requiring subsequent frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If frame rate interpolation is used to increase frame rate, then frame rate is improved, but latency increases significantly
Solution Approach 1:
Instead of interpolating future frames based on past and current frames (which requires waiting for future frames), the patent inverts the approach by extrapolating future frames based on past motion trends. This allows frame generation to proceed in the forward direction without waiting for subsequent frames, thereby reducing latency while maintaining improved frame rate.
Solution Approach 2:
The patent performs preliminary action by predicting and generating future frames in advance using motion extrapolation based on historical motion data from inertial sensors. By preparing frames ahead of time based on predicted device movement, the system reduces latency and enables smoother frame delivery without waiting for actual captured frames.
2Productivity
If higher frame rates are implemented in ER systems, then user experience smoothness is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary approach by using motion data from inertial measurement units as a bridge between captured frames. Instead of directly processing high-frequency camera data or complex scene analysis, the system uses simplified motion vectors from sensors to guide frame extrapolation, reducing computational complexity while achieving higher effective frame rates.
Solution Approach 2:
The patent replaces complex mechanical/image-based frame generation methods with a computational approach using motion extrapolation. Instead of relying on heavy image processing or mechanical frame capture at high rates, the system substitutes these with algorithmic frame prediction based on inertial sensor data, reducing overall system complexity.
Data Source
AI summary
In one implementation, a method of frame rate extrapolation is performed by a device including one or more processors, non-transitory memory, a scene camera, and a display. The method includes capturing, using the scene camera, an image of a scene. The method includes displaying, on the display, the image of the scene at a first time. The method includes generating an extrapolated image by transforming, using the one or more processors, the image of the scene based on movement of the device, wherein the extrapolated image includes a first area including a first plurality of pixels having respective first pixel values based on a single depth and a second area including a second plurality of pixels having respective second pixel values based on a plurality of depths. The method includes displaying, on the display, the extrapolated image at a second time after the first time.