Video Virtual Object Placement with 3D Triangulated Grids

The patent segments the sparse 3D point cloud into multiple local regions and performs plane estimation independently in each region. This allows the system to handle sparsity by treating each local area separately, where sufficient points may exist for reliable estimation, thus resolving the contradiction between simple processing and complete estimation.

The patent transitions from working directly with sparse 3D points to constructing a 2D depth map representation. This dimensional transformation allows the system to leverage 2D image space relationships and algorithms (like RANSAC in 2D) to infer 3D plane structures, overcoming the limitations of sparse 3D point distribution.

The patent applies different processing strategies to different regions of the scene. Planar regions undergo traditional plane estimation, while non-planar regions are identified and handled separately through depth map analysis and multiple hypothesis testing. This local differentiation allows the system to maintain simplicity where applicable while gaining versatility where needed.

The patent introduces a depth map as an intermediary representation between the 3D point cloud and plane estimation. This depth map serves as a mediator that preserves geometric information while enabling the use of 2D image processing techniques, thus bridging the gap between simple algorithms and complex scene geometry.

The patent performs preliminary plane estimation and depth map construction before virtual object placement. By pre-processing the scene to identify all available planes and surfaces, the system ensures that placement targets are ready in advance, maintaining fast placement speed while ensuring reliability through thorough preliminary analysis.

The patent implements a feedback mechanism where the system continuously refines plane estimates by comparing expected plane intersections with actual depth map data. This iterative refinement process ensures that plane information is complete and accurate before placement decisions are made, thereby guaranteeing placement success while maintaining efficiency through targeted refinement rather than exhaustive processing.