Neural Network Volumetric Video Depth Fidelity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Capturing accurate volumetric video is challenging due to difficulties in determining depth and color measurements, especially in scenarios like a boxing match where using multiple sensors is impractical due to logistical and cost constraints.
Innovation Solution
A method involving generating a high fidelity 3D model of a scene, aligning it with a perceived depth map, and training a neural network using both to increase the fidelity of captured depth and color maps, allowing for real-time processing and improved accuracy in less than 33 ms at 30 frames per second.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large number of sensors are used to capture the scene, then measurement precision of depth and color is improved, but device complexity and logistical challenges increase
Solution Approach 1:
A neural network is introduced as an intermediary between the limited sensor data and the required high-fidelity depth and color maps. The network learns to translate low-fidelity sensor inputs into high-fidelity outputs by training on synthetic data pairs, effectively mediating the information gap without requiring additional physical sensors.
Solution Approach 2:
The invention creates synthetic copies of training data through 3D models and rendering pipelines. Multiple virtual camera views are generated from a single 3D model, providing diverse training examples without requiring multiple physical cameras. This copying approach allows the system to learn from abundant synthetic data while using minimal physical sensors.
2Measurement precision
If a large number of sensors are placed around the scene, then measurement precision is improved, but ease of operation deteriorates due to logistical challenges
Solution Approach 1:
The neural network serves as an intermediary that processes data from a minimal sensor configuration (single or few cameras) to produce results equivalent to a dense sensor array. This eliminates the need for complex sensor deployment while maintaining measurement precision through learned transformations.
Solution Approach 2:
The system changes the parameter of sensor quantity from many to few, compensating for this reduction by transforming the data through a trained neural network. The network learns to infer depth and color information that would traditionally require multiple sensors, effectively changing the operational parameters while maintaining output quality.
3Productivity
If real-time processing is required at 30 frames per second, then productivity is improved, but measurement precision may deteriorate due to processing time constraints
Solution Approach 1:
The neural network is trained in advance on extensive synthetic training data before deployment. This preliminary training phase allows the network to learn complex transformations offline, so that during real-time operation at 30 fps, the network can quickly apply learned knowledge without requiring extensive processing time for each frame.
Solution Approach 2:
The system uses dynamic batch sizing and optimization techniques to adapt processing time to frame requirements. The neural network architecture is designed to process frames in a streamlined manner, adjusting computation dynamically to meet the 30 fps target while maintaining output fidelity through efficient inference pathways.
Data Source
AI summary
A first aspect of the invention provides a method of training a neural network for capturing volumetric video, comprising: generating a 3D model of a scene; using the 3D model to generate a high fidelity depth map; capturing a perceived depth map of the scene, having a field of view that is aligned with a field of view of the high fidelity depth map; and training the neural network based on the high fidelity depth map and the perceived depth map, wherein the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has.