AR Frame Reordering for Low-Latency Smartphone VR Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The need for specialized virtual reality hardware, such as headsets, creates accessibility barriers, and existing virtual reality applications on portable devices suffer from sensor latency and processing issues that can cause nausea and poor rendering, especially when using inexpensive viewers.
Innovation Solution
The solution involves reordering sensor data from gyroscopes, accelerometers, and augmented reality frames to align with capture times, using unscented Kalman filtering and reprojection techniques to improve data accuracy and reduce latency, and employing hand-tracking processes to enhance user interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If sensor data from gyroscopes, accelerometers, and augmented reality frames is processed in real-time order, then processing simplicity is maintained, but latency increases and data accuracy deteriorates
Solution Approach 1:
The system performs preliminary actions by capturing timestamps for all sensor data and AR frames, then reorders the data based on these timestamps before processing. This preliminary reordering ensures that data is processed in the correct temporal sequence, reducing latency and improving accuracy without requiring complex real-time synchronization mechanisms.
Solution Approach 2:
The patent introduces an intermediary data structure that stores both the sensor data and its corresponding timestamp. This intermediary structure acts as a mediator between data capture and processing, allowing the system to sort and reorder data based on timestamps while maintaining a clear processing pipeline. This approach manages complexity by separating the concerns of data collection, timestamping, reordering, and processing.
2Measurement precision
If augmented reality frames are processed to provide accurate position and orientation data, then virtual reality experience quality improves, but processing time increases causing nausea
Solution Approach 1:
The system performs preliminary timestamping of AR frames and sensor data as they are captured. By recording the exact capture time beforehand, the system can later reorder the data correctly without requiring complex real-time synchronization. This preliminary action maintains measurement precision while enabling efficient batch processing that reduces overall latency.
Solution Approach 2:
The patent implements dynamic reordering of sensor data and AR frames based on their timestamps. Rather than processing data in a fixed sequential order, the system dynamically adjusts the processing order to match the actual temporal sequence of events. This dynamic approach ensures that the most recent data is processed first, reducing latency and preventing nausea while maintaining accurate position and orientation measurements.
3Manufacturing precision
If specialized virtual reality hardware is used, then rendering quality and user experience improve, but accessibility and cost decrease
Solution Approach 1:
The patent makes the portable device universal by enabling it to perform both augmented reality and virtual reality functions using the same hardware components (camera, gyroscopes, accelerometers, display). The system achieves this by processing AR frame data and sensor data together to create VR experiences, eliminating the need for specialized VR hardware and improving accessibility while maintaining rendering quality.
Solution Approach 2:
The system changes the operational parameters of the portable device to enable VR functionality. By adjusting how the existing camera and sensor data are processed and combined, the system transforms the device's capabilities without requiring hardware changes. This parameter-based approach allows standard portable devices to deliver VR-quality experiences, improving accessibility while maintaining rendering quality.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach provides a more immersive and nausea-free virtual reality experience on portable devices by aligning sensor data with capture times, reducing latency, and optimizing rendering processes, making virtual reality more accessible.
Implementation Method 1
The service may use an unscented Kalman filter to order the collection of sensor inputs
Data Source
AI summary
The present technology combines low latency sensor data from gyroscopes and accelerometers with accurate but higher latency data from AR frames by reordering a received series of data collected from the gyroscopes, accelerometers, and AR frames so that the AR frame is placed in the series of data earlier in time than it was received—to a time that corresponds with gyroscope and accelerometer data corresponding to a position and time which the AR frame capture was initiated. The correctly ordered data can then be used to project where the user will be looking at a future time so that the virtual reality application can render a scene that corresponds with the user's expectations. In this way, a smartphone can be used to provide a good virtual reality experience when used with a simple viewer, thus making virtual reality experiences more accessible.