Remote Rendering Prediction for Low-Latency AR and VR Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote rendering systems face challenges in reducing processing time, storage requirements, and network delays, particularly in AR and VR applications, leading to user perception issues like VR sickness due to deviations in line-of-sight direction and image rendering delays.
Innovation Solution
A remote rendering system that predicts future sensor states to pre-render images for anticipated positions, utilizing a master and slave server architecture to optimize resource utilization and reduce delays by selecting optimal servers based on proximity and available bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If pre-rendering is performed for all user viewpoints, then rendering delay is reduced, but processing time and storage requirements increase enormously
Solution Approach 1:
The patent applies local quality by pre-rendering images only for specific predicted viewpoints based on user behavior patterns, rather than all possible viewpoints. The system identifies and prioritizes rendering for likely user positions and orientations, allocating storage and processing resources locally to the most probable viewing angles, thus reducing overall storage requirements while maintaining low rendering delay for actual user experiences.
Solution Approach 2:
The system performs preliminary rendering actions by predicting future user viewpoints based on current sensor data and pre-rendering images for those predicted positions before the user actually reaches them. This anticipatory approach reduces the perceived rendering delay while avoiding the need to pre-render all possible viewpoints, thereby optimizing the balance between rendering speed and storage requirements.
2Loss of time
If pre-rendering is performed for all user viewpoints, then rendering delay is reduced, but processing time increases enormously
Solution Approach 1:
The patent applies local quality by concentrating processing resources on rendering images for predicted viewpoints rather than distributing resources across all possible viewpoints. By identifying the specific subset of viewpoints the user is likely to encounter and prioritizing rendering for those positions only, the system reduces total processing time while maintaining low rendering delay for the actual user experience.
Solution Approach 2:
The system performs preliminary rendering actions anticipatorily by predicting future user viewpoints based on sensor data and pre-rendering images for those positions before they are needed. This selective preliminary action reduces the critical rendering delay for user-facing images while avoiding the enormous processing time required to pre-render all possible viewpoints.
3Loss of time
If network delay and encoding/decoding delay are reduced through remote rendering, then user experience improves, but resource shortage on remote server occurs
Solution Approach 1:
The system performs preliminary rendering actions by predicting future user viewpoints and pre-rendering images for those positions before they are actually needed. This anticipatory rendering reduces the critical network and encoding delays during user interaction, while the selective nature of predicting only likely viewpoints (rather than all possibilities) helps manage server resource consumption more efficiently.
4Use of energy by moving object
If rendering is performed in local environment, then resource shortage on server is avoided, but motion-to-photon latency increases
Solution Approach 1:
The system performs preliminary rendering actions by predicting future user viewpoints and pre-rendering images for those positions before they are actually needed. This anticipatory approach reduces the motion-to-photon latency that would otherwise occur with local rendering, while the selective prediction of only likely viewpoints keeps server resource utilization manageable.
Data Source
AI summary
The present disclosure includes a remote rendering system, an image processing method, a server device, and a program The remote rendering system includes a terminal having a sensor, and a server, the terminal transmitting information about a sensor state to the server, and the server transmitting a rendered image corresponding to the sensor state to the terminal. The terminal transmits information about the current sensor state to the server, and the server predicts a plurality of possibilities of the sensor state to be expected in an immediate future based on the received information about the current sensor state. The server performs rendering corresponding to the predicted possibilities of the sensor state, and stores a plurality of the rendered images generated by the rendering.