Split AR Rendering Using Predicted Pose for Low-Power Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital video devices face challenges in efficiently managing the rendering of augmented reality media data, particularly in devices with limited processing power, leading to suboptimal user experiences and excessive power consumption.
Innovation Solution
Implementing a split rendering approach where a split rendering server processes and renders at least part of the augmented reality media data, streaming the rendered images to a display device like AR glasses or a head-mounted display, while conveying pose information using Real-time Transport Protocol (RTP) header extensions to ensure accurate display alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If rendering is performed locally on the display device, then rendering control is flexible and responsive, but processing power requirements increase and power consumption rises
Solution Approach 1:
The rendering process is segmented into two parts: complex rendering computations are performed on a remote server, while the local display device handles pose tracking, prediction, and final image composition. This division allows the display device to use less processing power and consume less energy while maintaining rendering control.
Solution Approach 2:
A split rendering server acts as an intermediary between the content source and the display device. The server receives rendering requests with pose information, performs the computationally intensive rendering, and returns rendered images to the display device, thereby reducing the processing burden on the display device.
2Power
If rendering is performed on a remote server, then processing demands on the display device are reduced, but latency in image delivery may increase
Solution Approach 1:
The display device performs pose prediction in advance to estimate future user head positions and orientations. By sending these predicted pose information to the server before rendering is needed, the server can begin rendering computations earlier, reducing the overall latency of the rendering pipeline.
Solution Approach 2:
The system implements a feedback loop where the display device continuously tracks actual user pose, compares it with predicted pose, and uses this information to refine future predictions. This feedback mechanism improves prediction accuracy over time, ensuring that rendered images remain aligned with the user's actual viewpoint despite the remote rendering delay.
3Measurement precision
If pose information is transmitted continuously, then rendering accuracy is maintained, but network bandwidth consumption increases
Solution Approach 1:
Instead of transmitting continuous pose data, the system transmits predicted pose information at strategically chosen moments before rendering is needed. This preliminary transmission of essential pose data maintains rendering accuracy while significantly reducing the frequency and volume of network transmissions.
Solution Approach 2:
The display device creates a local copy of the pose tracking functionality by implementing pose prediction algorithms. Rather than relying entirely on continuous server-side pose processing, the local prediction copy reduces the need for frequent pose data exchanges over the network.
Data Source
AI summary
An example device for presenting split-rendered media data includes a memory configured to store media data; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: send pose information representing a predicted pose of a user at a first future time to a split rendering server; receive an at least partially rendered image for the first future time and data associating the pose information with the at least partially rendered image from the split rendering server; and at a second future time, present a rendered image based on the partially rendered image.