Real-Time 3D Geometry Generation Using Hierarchical Ray Marching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for real-time 3D imaging are computationally expensive and require complex setups or prior templates, struggling to handle changing scene topologies and provide acceptable quality with low computational complexity.
Innovation Solution
A method involving a calibrated RGB camera pair for stereo reconstruction, using ray marching, uniqueness criteria, regularization, and interpolation to generate 3D geometry efficiently, without the need for templates or specific lighting, and capable of handling dynamic scenes with changing topologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If full end-to-end free-viewpoint video pipeline with Poisson Surface Reconstruction is used, then high-quality reconstruction is achieved, but processing time becomes extremely long (28.2 minutes per frame)
Solution Approach 1:
The patent segments the 3D reconstruction process into multiple hierarchical levels of detail (LOD). Instead of computing full high-resolution geometry for every pixel, the system performs ray marching at multiple resolution levels, progressively refining geometry only where needed. This segmentation of computational effort across resolution levels dramatically reduces processing time while maintaining reconstruction quality in visible areas.
Solution Approach 2:
The patent applies partial action by computing geometry only for pixels that actually contribute to the final image. Using depth buffering and occlusion culling, the system avoids performing expensive ray marching operations for pixels that will be occluded or not displayed. This selective computation maintains reconstruction quality for visible surfaces while eliminating wasted computational effort on invisible areas.
2Productivity
If template-based approach with custom-built RGBD camera is used, then real-time high-quality reconstruction is achieved, but the method requires a fixed template that cannot handle changing scene topologies
Solution Approach 1:
The patent implements dynamics by making the 3D model adaptable and mutable during runtime. Instead of using a fixed template, the system continuously updates the mesh geometry based on incoming depth data from RGBD cameras. The mesh topology can change dynamically to accommodate new objects or scene modifications, while maintaining real-time performance through efficient update strategies that only modify affected regions.
Solution Approach 2:
The patent changes parameters by allowing the mesh properties (vertices, faces, normals) to be dynamically modified based on incoming sensor data. The system adjusts mesh density, resolution, and topology parameters adaptively according to scene content and camera viewpoints, enabling handling of changing scene topologies while maintaining real-time performance through controlled parameter updates.
3Adaptability or versatility
If Kinect Fusion with non-rigid volumetric fusion is used, then dynamic scenes are handled without explicit template, but the method still has problems with changing scene topologies due to reliance on reference surface model
Solution Approach 1:
The patent uses copying by creating and maintaining a dynamic 3D mesh model that is continuously updated from multiple RGBD camera views. Instead of relying on a single reference surface model, the system copies and integrates geometry information from multiple sources, allowing the mesh to adapt to changing scene topologies while maintaining reconstruction accuracy through multi-view geometric constraints.
4Manufacturing precision
If Fusion4D with active stereo approach is used, then multiple views are handled in real time with high quality, but the system becomes computationally expensive requiring dedicated machines with high-end GPUs
Solution Approach 1:
The patent segments the computational workload by processing different image regions at different resolution levels. Important regions (visible surfaces, high-curvature areas) receive full-resolution processing, while less important regions use lower resolution. This segmentation of computational effort maintains reconstruction quality for critical areas while significantly reducing overall energy consumption and hardware requirements.
Solution Approach 2:
The patent applies local quality by allocating computational resources non-uniformly across the image domain. High-quality ray marching and geometry computation are performed only in regions that contribute significantly to the final image quality, while other regions use simplified processing. This local differentiation maintains overall reconstruction quality while reducing total computational resource consumption.
Data Source
AI summary
The present invention relates to a method of real-time generation of a 3D geometry of an object. The method comprises the steps of calibrating at least one RGB camera pair arranged to provide images of the object, receiving input images of the object from the at least on RGB camera pair, and performing a stereo reconstruction in a first hierarchical level using the input images. The stereo reconstruction step further comprises the steps of a) performing a ray marching operation in a first resolution on the input images to determine geometry positions along each view ray of the images; b) applying a uniqueness criterion to the geometry positions; c) determining a normal for each geometry position; d) performing a regularization operation based on the geometry positions and the respective normal, providing updated geometry positions; and e) performing an interpolation operation on the updated geometry positions and respective normal. The method further comprises repeating steps a) and c)-e) in at least one iteration in at least one ascending hierarchical level, wherein the resolution in the ray marching operation is doubled for each iteration, resulting in a geometry buffer for each of the at least one camera pair. The present invention further relates to a 3D image generation device and a system comprising such device.