Client-Server 3D Scene Streaming with Viewpoint Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing client-server communication in streaming of visual content, particularly for volumetric video objects, faces challenges in achieving a balance between low data rate and realistic user experience, especially for devices with limited processing power, such as smart glasses, due to the need for high computational resources and network latency.
Innovation Solution
A message system is implemented that allows the client to provide feedback to the server about viewpoint mismatches, latency, user-object interactions, rendering processes, and 3D scene properties, enabling the server to adapt the visual content transmission and rendering accordingly, optimizing data transmission and quality based on client capabilities and user behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If visual content is streamed to resource-constrained devices like smart glasses, then device portability and accessibility are improved, but processing power and computational resources become insufficient for rendering complex volumetric video objects
Solution Approach 1:
The patent extracts the rendering function from the client device and relocates it to a remote server. The server renders volumetric video objects from multiple viewpoints and transmits pre-rendered 2D video streams to the client, eliminating the need for complex local rendering hardware while maintaining immersive visual content delivery
Solution Approach 2:
The patent introduces an intermediary server that acts as a computational bridge between the volumetric video source and the resource-constrained client device. The server performs viewpoint estimation, rendering, and video stream generation, mediating the complex processing requirements between the content source and the portable display device
2Device complexity
If server-side rendering is implemented to reduce client processing requirements, then client device complexity is reduced, but network latency and data transmission requirements increase
Solution Approach 1:
The patent applies preliminary action by having the server pre-render video streams from multiple predetermined viewpoints before transmission. This allows the client to quickly switch between pre-computed views without requiring real-time rendering, significantly reducing perceived latency while maintaining server-side rendering benefits
Solution Approach 2:
The patent implements dynamic viewpoint switching where the client estimates the user's current viewpoint and requests appropriate pre-rendered streams from the server. The system dynamically adapts between different pre-computed views based on user movement, balancing rendering complexity with responsive interaction
3Reliability
If high-quality volumetric video objects are transmitted to ensure realistic user experience, then visual quality is improved, but data rate and network bandwidth requirements increase
Solution Approach 1:
The patent applies local quality by rendering different regions of the visual field at different resolutions and qualities. The server prioritizes high-quality rendering for the foveal region (center of vision) while using lower resolution for peripheral regions, maintaining realistic visual experience while significantly reducing overall data transmission requirements
Solution Approach 2:
The patent segments the volumetric video content into multiple 2D video streams corresponding to different viewpoint regions. Each segment is independently encoded and transmitted, allowing selective transmission of only those segments currently needed for the user's viewpoint, reducing redundant data transmission
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A concept for transmitting information from a client to a server is described, the client being for streaming visual content from the server and for integrating the visual content into a 3D scene of a user application, and the client being for obtaining a rendered view of the 3D scene.