Cloud-Hosted AR Rendering with Client-Side Overlay for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Streamed augmented reality experiences face significant latency issues due to the time it takes for video feed and user inputs to be transmitted to the cloud, processed, and then streamed back to the client device, resulting in an unpleasant and laggy experience.
Innovation Solution
The technology runs the main portion of the augmented reality code in a cloud-hosted service, streaming graphical components and rendering instructions to the client device, which performs the final rendering and overlaying of content onto its video stream, using deltas relative to a previous state to reduce data communication and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If augmented reality code is run entirely in the cloud with full video feed transmission, then processing power and experience library access are improved, but latency and network dependency worsen
Solution Approach 1:
The patent segments the augmented reality processing into two parts: complex scene understanding and object recognition are performed in the cloud using powerful servers, while simple rendering and overlay operations are performed locally on the client device. This segmentation allows the system to leverage cloud computing power for intensive tasks while maintaining low latency for time-critical rendering operations.
2Measurement precision
If full video feed is transmitted to the cloud for processing, then processing accuracy is improved, but network bandwidth consumption and latency worsen
Solution Approach 1:
The patent extracts and transmits only the essential data elements needed for cloud processing - specifically keyframe images and device pose information - rather than transmitting the entire continuous video feed. This extraction approach maintains processing accuracy by providing sufficient data for scene understanding while dramatically reducing network bandwidth consumption.
3Productivity
If cloud processing is used for all AR operations, then experience quality and CPU capabilities are improved, but perceived latency and user experience smoothness worsen
Solution Approach 1:
The patent implements preliminary action by having the client device perform rendering operations locally using previously received scene understanding data from the cloud. This allows the device to pre-render frames and maintain a smooth visual experience even when cloud processing is still working on the latest input data, effectively hiding the latency of cloud operations.
Data Source
AI summary
A technology that streams graphical components and rendering instructions to a client device, for the client device to perform the final rendering and overlaying of that content onto the client's video stream based on the client's most recent tracking of the device's position and orientation. A client device sends a request for augmented reality drawing data to a network device. In response, the network device generates augmented reality drawing data, which can be augmented reality change data based on the augmented reality information and previous client render state information, and sends the augmented reality drawing data to the client device. The client device receives the augmented reality drawing data and renders a visible representation of an augmented reality scene comprising overlaying augmented reality graphics over a current video scene obtained from a camera of the client device.