Volumetric Video AR Mapping With Remote Rendering and Occlusion Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in distributing volumetric video objects to user devices due to hardware limitations, rendering complex scenes with multiple objects, and high transmission bitrate, resulting in suboptimal integration of 2D video into spatial 3D AR scenes with issues like occlusions and collisions.
Innovation Solution
A method and user device for augmenting a real-world environment with a volumetric video object by determining current and desired poses, rendering the object remotely, and mapping it onto a planar surface on the user device using block-based coding and transparency maps to ensure perspective-free projection and efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If volumetric video is rendered locally on user devices, then rendering quality and interactivity are improved, but hardware complexity and computational requirements increase beyond current mobile device capabilities
Solution Approach 1:
A remote rendering server acts as an intermediary between the volumetric video source and the user device. The server performs the computationally intensive volumetric video rendering and transmits only the final 2D rendered views to the user device, eliminating the need for complex local hardware while maintaining high rendering quality
Solution Approach 2:
The patent replaces the mechanical/computational rendering system on mobile devices with a cloud-based rendering system. Instead of using local GPU resources, the system uses remote server resources to render volumetric video, substituting local computational mechanics with network-based computational mechanics
2Adaptability or versatility
If complex scenes with multiple volumetric objects are rendered, then scene complexity and visual richness are improved, but rendering time and computational load increase significantly
Solution Approach 1:
The remote rendering server automatically handles the complexity of rendering multiple volumetric objects in a scene without requiring user intervention or local device processing. The server independently manages scene composition, object rendering, and view synthesis, allowing complex scenes to be rendered efficiently in the cloud
3Manufacturing precision
If high-resolution volumetric video is transmitted over the network, then video quality is improved, but transmission bitrate and network bandwidth requirements increase to tens of Mbit/s
Solution Approach 1:
The patent extracts only the essential rendered view information from the volumetric video data and transmits it to the user device. Instead of transmitting all the raw volumetric data that would require high bitrates, only the final 2D rendered frames are transmitted, significantly reducing network bandwidth requirements while maintaining video quality
Solution Approach 2:
Instead of transmitting volumetric data and rendering it locally (which would require high bandwidth), the system inverts the approach by rendering first and then transmitting only the rendered 2D views. This inversion of the processing sequence dramatically reduces the amount of data that needs to be transmitted over the network
4Ease of operation
If 2D video is integrated into 3D AR scenes without volumetric information, then implementation simplicity is improved, but spatial integration quality deteriorates due to occlusions and collisions
Solution Approach 1:
The system creates a 2D copy of the volumetric video object from multiple viewpoint images and transmits this rendered view to the user device. This 2D copy maintains the appearance of the volumetric object from the user's current viewpoint, enabling simple 2D integration while preserving the visual realism of the original volumetric scene
Data Source
Figure 1~3
Figure 4
AI summary
The present invention relates to a method of augmenting a view (2) of a real- world environment (3) with a view (4) of a volumetric video object (5) on a user device (10). The method comprises determining a current pose information (CPI) indicating a current pose of the view (2) of the real-world environment (3) and a desired pose of the volumetric video object (5) in the real-world environment (3). The method further comprises sending the current pose information (CPI) to a remote server (20). The method further comprises receiving a rendered view (4) of the volumetric video object (5) that has been rendered in accordance with the current pose information (CPI) from the remote server (20). The method further comprises augmenting the view of the real-world environment (3) by at least mapping the rendered view (4) of the volumetric video object (5) onto a planar mapping surface (6, 6') arranged according to the desired position of the volumetric video object (5).