Per-Object Bundle Adjustment for Volumetric Video Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for generating real-time volumetric video in high-profile events, such as sporting events, require high computational power and struggle to maintain high quality and immersion due to the complexity of 3D reconstruction from multiple cameras, especially in dynamic scenes with both static and non-rigid moving objects.
Innovation Solution
The implementation of a system that uses per-object calibration modeling, involving a dense 3D reconstruction module and a per-object real-time calibration module to generate virtual views within a scene by constructing and solving separate bundle adjustment models for each object, optimizing camera parameters and point cloud points concurrently, and alternating between resection and intersection techniques for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple cameras are used for 3D reconstruction to provide immersive media experience, then the quality and immersion of the volumetric content is improved, but the computational power required increases significantly
Solution Approach 1:
The patent divides the scene into multiple separate 3D objects and processes each object independently with its own bundle adjustment model. This segmentation allows parallel processing of multiple objects simultaneously, reducing the overall computational burden while maintaining high reconstruction quality for each object
Solution Approach 2:
The patent applies bundle adjustment selectively only to dynamic objects that require high precision, rather than processing all objects in the scene uniformly. This partial action approach focuses computational resources on critical elements while reducing overall computation time and power requirements
2Reliability
If real-time volumetric video is generated from multiple cameras, then the immersive user experience is improved, but the processing time and latency increase
Solution Approach 1:
By segmenting the scene into separate 3D objects and processing them independently, the system can process multiple objects in parallel rather than sequentially. This parallel processing significantly reduces total processing time and latency while maintaining real-time capability
Solution Approach 2:
The system performs preliminary calibration and creates initial 3D models of objects before the main volumetric video generation process. This preliminary action prepares the data structure in advance, enabling faster real-time processing and reducing latency during actual video generation
3Measurement precision
If separate bundle adjustment models are constructed for each 3D object, then the 3D reconstruction precision is improved, but the device complexity increases
Solution Approach 1:
The patent creates separate bundle adjustment models for each 3D object, which simplifies the mathematical computations for each individual object compared to a single global model. This segmentation makes the system more manageable and easier to implement despite processing multiple objects
Solution Approach 2:
The patent uses a universal bundle adjustment framework that can be applied to multiple different objects with the same algorithmic approach. This multi-functionality reduces system complexity by reusing the same core processing logic across all objects rather than requiring object-specific custom algorithms
Data Source
AI summary
Techniques related to improved continuous local 3D reconstruction refinement are discussed. Such techniques include constructing and solving per 3D object adjustment models in real time to generate a point cloud and/or updated camera parameters for each object adjustment model.