3D Scene Rendering via Limited Video Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for rendering panoramic videos in Extended Reality (XR) face challenges such as coarse viewpoint distribution, high computational burden, and bandwidth requirements due to the need to stream entire videos for changes in viewing position, which can lead to latency and resource overload.
Innovation Solution
A method where a client device receives video streams of objects from a limited set of viewing angles based on relative position, allowing for dynamic adjustment of perspective without requesting new video streams, reducing bandwidth and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of predefined viewpoints is increased to provide finer granularity for user movement, then the sense of free movement is improved, but the number of omnidirectional video cameras required increases greatly, posing practical challenges
Solution Approach 1:
The scene is segmented into multiple spherical video representations captured at different spatial positions. Instead of using a single omnidirectional camera or increasing the number of cameras, the system divides the scene into multiple viewpoints, each captured by a separate omnidirectional camera. This segmentation allows the system to provide fine-grained viewpoint coverage while keeping the number of cameras manageable.
Solution Approach 2:
A synthesis device acts as an intermediary between the captured spherical videos and the client device. This synthesis device generates intermediate viewpoints that are not directly captured by cameras but are computationally synthesized from the captured videos. This intermediary process enables fine-grained viewpoint provision without requiring a proportional increase in the number of physical cameras.
2Adaptability or versatility
If viewpoint synthesis is performed at the client device to provide fine-grained viewpoints, then the sense of free movement is improved, but the computational burden on the client device increases significantly
Solution Approach 1:
Instead of performing viewpoint synthesis at the client device (as in conventional approaches), the system inverts the synthesis location to a dedicated synthesis device on the server side. The client device only receives pre-synthesized spherical videos and performs simple rendering operations. This inversion shifts the heavy computational burden from the client device to the server-side synthesis device.
Solution Approach 2:
The synthesis device creates multiple copies of spherical video content at different viewpoints and spatial positions. These synthesized video copies are then transmitted to the client device, which only needs to perform lightweight rendering operations to display the appropriate view. This copying approach eliminates the need for the client device to perform computationally intensive viewpoint synthesis.
3Speed
If multiple video streams are requested simultaneously to avoid latency during movement, then the responsiveness to viewing position changes is improved, but the bandwidth requirement and resource burden increase greatly
Solution Approach 1:
The system dynamically adapts the video streaming based on the client device's current viewing position and movement state. Instead of continuously streaming multiple video streams, the system streams only the spherical video content that is currently needed for the client's viewpoint. This dynamic adaptation reduces bandwidth consumption while maintaining responsiveness to viewing position changes.
Solution Approach 2:
The client device performs self-service by autonomously determining which spherical video content to request based on its current viewing position and orientation. The client device calculates the appropriate spherical video representation needed and requests only that specific content from the server, without requiring the server to proactively send multiple streams. This self-service approach minimizes bandwidth usage while ensuring the client receives the necessary content in a timely manner.
Data Source
AI summary
A client device may be enabled to render a three-dimensional [3D] scene comprising one or more objects. At a server system, a video-based representation of an object may be streamed as one or more video streams to the client device. At the client device, the scene may be rendered from a viewing position within the scene to obtain a rendered view of the scene. Furthermore, a relative position between the viewing position and a position of the object in the scene may be determined. At the server system, the one or more video streams may be generated to show the object from a limited set of viewing angles, wherein the limited set of viewing angles is selected based on the relative position. At the client device, a viewing angle may be selected from the limited set of viewing angles and the video-based representation of the object may be placed at said selected viewing angle in the scene.