Remote AR Rendering Latency Reduction via Metadata-Driven Compositing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current VR and AR technologies face challenges in providing immersive experiences due to the limited computational power and battery capacity of head-mounted devices, leading to latency issues in rendering and augmenting multimedia content.
Innovation Solution
A processing flow for AR services is defined, involving extra processing of rendered 2D frames before transmission and additional processing on the AR device for compositing, along with metadata transmission to ensure accurate augmentation and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If heavy computational processing is performed on head-mounted AR devices, then rendering quality and immersion experience are improved, but device complexity and power consumption increase significantly
Solution Approach 1:
The patent extracts the heavy computational processing function from the head-mounted AR device and relocates it to a remote server. The AR device only performs lightweight tasks such as capturing real-world images, tracking user position and orientation, and compositing the received virtual content with the real-world background. This extraction principle resolves the contradiction by maintaining high rendering quality through remote processing while keeping the AR device simple and low-power.
2Productivity
If heavy computational processing is performed on head-mounted AR devices, then virtual content generation capability is improved, but battery capacity requirements and heat generation increase
Solution Approach 1:
The patent extracts the energy-intensive virtual content generation process from the battery-powered AR device and performs it on a remote server with unlimited power supply. The AR device receives pre-generated virtual content and only performs minimal processing to composite it with the real-world environment, thereby maintaining high productivity while dramatically reducing battery capacity requirements.
Solution Approach 2:
The patent introduces a communication network as an intermediary between the AR device and the remote server. This intermediary enables the transfer of virtual content from the server to the device without requiring the device to perform heavy local computation, thus reducing energy consumption while maintaining content generation capability.
3Reliability
If 3D virtual objects are augmented into the user's environment in real-time, then immersion experience is improved, but latency in rendering and transmission increases
Solution Approach 1:
The patent applies preliminary action by pre-generating virtual content on the remote server before it is needed by the AR device. The server prepares 3D models, textures, and other virtual elements in advance, so that when the AR device requests content, it is already available for rapid transmission and compositing. This reduces rendering latency while maintaining high immersion quality.
Data Source
AI summary
A method for augmented reality (AR) remote rendering process performed by a remote rendering device. The method includes performing 3D scene compositing based on a live geometry information and an anchor information; receiving, from an AR device, a pose information; rendering 2D frame of the composited 3D scene based on the pose information; performing 2D frame processing on the rendered 2D frame; creating a first metadata associated with the 2D frame processing, a second metadata associated with the rendered 2D frame and a third metadata associated with AR frame compositing of the rendered 2D frame; and transmitting, to the AR device, the rendered 2D frame, the first metadata, the second metadata and the third metadata.