Lightweight View-Dependent Rendering for Mobile Volumetric Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Volumetric video data is excessively large and computationally intensive, making it difficult to render on mobile devices without significant processing power, as it includes three-dimensional point clouds and video data from multiple camera views, which is not manageable for real-time or mobile applications.
Innovation Solution
A lightweight view-dependent rendering system that restricts camera motion to avoid parallax rendering, uses billboards to represent moving regions, and applies a matte to separate foreground from background, allowing for 2D rescaling and translation, eliminating the need for depth-based interpolation and compression of point clouds, and uses 2D video compression with caching for faster transitions between camera views.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional volumetric video rendering is used with full point cloud data and multiple camera views, then rendering quality and view dependency are improved, but data processing requirements and power consumption increase excessively for mobile devices
Solution Approach 1:
The system segments the volumetric video data by separating foreground moving regions from the background. Only the moving regions are tracked and rendered with full detail, while the background is handled separately. This segmentation reduces the amount of data that needs to be processed in real-time, thereby reducing power consumption while maintaining rendering quality for the important foreground elements.
Solution Approach 2:
The invention extracts and isolates the moving regions from the complete volumetric scene. By taking out only the necessary foreground elements that require detailed rendering and separating them from the background, the system avoids processing the entire point cloud dataset, thus reducing computational load and power consumption on mobile devices while preserving rendering quality for the extracted regions.
2Adaptability or versatility
If full volumetric data including point clouds from multiple cameras is processed, then view dependency and rendering accuracy are improved, but device complexity and processing requirements become intractable for mobile applications
Solution Approach 1:
The system segments volumetric video into moving regions and background, processing only the moving regions with full view dependency. This segmentation allows the mobile device to handle view-dependent rendering for critical foreground elements without needing to process the complete volumetric dataset, thereby reducing device complexity while maintaining adaptability for the important regions.
Solution Approach 2:
The invention creates simplified representations (copies) of the volumetric data for moving regions using billboards and tracked positions. Instead of processing the full high-fidelity point cloud data from multiple cameras, the system uses these simplified copies that capture the essential view-dependent information, reducing processing requirements while maintaining adequate visual fidelity for mobile devices.
3Productivity
If real-time rendering of complete volumetric video is attempted on mobile devices, then rendering speed and user experience are improved, but data processing requirements exceed mobile device capabilities
Solution Approach 1:
The system segments the volumetric scene into moving foreground regions and background, processing only the moving regions in real-time. This segmentation dramatically reduces the quantity of data that must be processed for real-time rendering, enabling mobile devices to achieve acceptable rendering speeds by focusing computational resources only on the dynamic elements that require frequent updates.
Solution Approach 2:
The invention extracts and processes only the essential moving regions from the complete volumetric dataset. By taking out and separately handling the foreground elements that change over time, the system reduces data processing requirements to a manageable level for mobile devices while maintaining real-time rendering capability for the extracted regions, thereby achieving acceptable productivity.
4Measurement precision
If depth-based interpolation and point cloud compression are used, then rendering accuracy is improved, but processing complexity and power consumption increase for mobile devices
Solution Approach 1:
The system uses simplified billboard copies positioned at tracked locations to represent moving regions, eliminating the need for complex depth-based interpolation and point cloud compression algorithms. These billboard copies provide adequate visual representation without requiring the sophisticated processing pipelines needed for high-precision depth reconstruction, thereby reducing processing complexity while maintaining acceptable rendering accuracy for mobile devices.
Data Source
AI summary
An example system for lightweight view dependent rendering is described herein. An example mobile device includes a display, instructions, and processor circuitry to execute instructions to extract a moving region within a first set of frames of volumetric video content captured by a plurality of real-world cameras facing a scene at different angles. The processor circuitry to generate a first billboard based on a cropped area of the frames in the first set of frames, the cropped area corresponding to the first moving region. The processor circuitry to, in response to user selection of a first view of the scene, cause presentation of the first billboard on the display. The processor circuitry to, in response to a change from the first view to a second view of the scene, cause a second billboard to replace the first billboard on the display.