Latency-resilient cloud rendering using view-dependent image synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional cloud-rendering approaches for Augmented-Reality (AR) and Virtual-Reality (VR) content result in latency issues due to network transmission delays, causing lag, especially when dealing with complex geometry and poor network conditions, as the server's viewpoint may differ from the user's by the time the video is displayed.
Innovation Solution
A method where a server performs computationally expensive rendering tasks and encodes 3D object artifacts, such as simplified geometry and RGB data, which are then sent to user devices to locally reconstruct images based on the user's up-to-date viewpoint, using a view hierarchy and machine learning techniques like neural radiance fields to generate view-dependent images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional cloud rendering is used where the server performs all rendering tasks and sends the final video stream to the client device, then the rendering quality can be high, but the latency increases significantly due to network transmission delays
Solution Approach 1:
The patent segments the rendering pipeline into two parts: (1) the server performs computationally expensive tasks including generating 3D scene representations, rendering depth buffers, and preparing view-dependent textures; (2) the client device performs lightweight reconstruction tasks using received data and local viewpoint information. This segmentation allows high-quality rendering while reducing network transmission requirements and latency.
Solution Approach 2:
The server performs preliminary rendering actions by pre-computing depth buffers, geometric representations, and view-dependent textures before the client needs them. These pre-computed data structures are transmitted to the client, which then performs rapid local reconstruction based on the user's current viewpoint, eliminating the need to wait for full re-rendering from the server.
2Device complexity
If the server renders video based on its current viewpoint and sends it to the user device, then the rendering process is simplified, but the viewpoint difference between server and user causes lag and reduces immersion
Solution Approach 1:
The system implements feedback by continuously tracking the user's device orientation and position, then using this information to determine which pre-rendered views or view-dependent textures to select and blend. This feedback loop ensures the displayed content always corresponds to the user's current viewpoint, eliminating lag and maintaining immersion.
Solution Approach 2:
The rendering system transitions from static pre-rendered views to dynamic view synthesis. The server provides view-dependent textures and depth information that enable the client to dynamically reconstruct images matching the user's real-time viewpoint, making the rendering adaptive rather than fixed.
3Manufacturing precision
If cloud rendering is used for complex geometry, then the rendering quality improves, but the network bandwidth requirements increase and performance deteriorates under poor network conditions
Solution Approach 1:
The patent extracts only the essential data elements needed for high-quality rendering—specifically depth buffers, simplified geometric representations, and view-dependent textures—from the full rendering pipeline. By transmitting only these compressed data structures rather than complete video streams, the system achieves high rendering quality for complex geometry while minimizing network bandwidth consumption.
Data Source
AI summary
A method includes the steps of receiving training data comprising images of an object and associated camera poses from which the images are captured, training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object viewed from the given viewpoint, generating, for each of predetermined viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model, receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object, selecting one or more of the predetermined viewpoints based on the desired viewpoint, and sending, to the client device, the view-dependent images associated with the selected one or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.