3D-Consistent Plane Embeddings for Multi-View Plane Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D plane estimation methods, particularly in multi-image settings, struggle to accurately identify and track planar regions across frames, limiting their application in robotics and augmented reality.
Innovation Solution
A method for determining consistent plane embeddings from a sequence of posed RGB images by mapping vertices to a multi-view consistent plane embedding space and clustering them using geometrical priors, generating an initial 3D scene geometry through a truncated signed distance function (TSDF) and extracting a 3D mesh, followed by clustering vertices into 3D plane instances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bottom-up approaches map pixels into embeddings and group into planes via clustering, then plane estimation can be performed, but accuracy in tracking planar regions across multiple frames is poor
Solution Approach 1:
The patent transforms 2D image embeddings into 3D-consistent embeddings by adding a third dimension that represents spatial coherence across multiple views. This dimensional extension enables the system to maintain consistent plane representations across frames while preserving the clustering capability, thereby resolving the contradiction between plane estimation accuracy and tracking reliability.
Solution Approach 2:
The patent implements a feedback mechanism where plane embeddings are refined iteratively using geometric constraints from multiple views. The system continuously adjusts embeddings based on consistency checks across frames, improving both the accuracy of plane estimation and the reliability of tracking over time.
2Measurement precision
If vision transformers are used for single-image plane estimation, then state-of-the-art results are achieved, but extension to multi-image settings is limited
Solution Approach 1:
The patent extends the vision transformer architecture to serve multiple functions: single-image plane estimation, multi-image consistency learning, and 3D scene reconstruction. By designing the model to process sequences of images and learn temporal-spatial consistency, it achieves universality across different task settings while maintaining the state-of-the-art accuracy of the original single-image approach.
3Manufacturing precision
If depth maps are fused into TSDF for 3D mesh extraction, then initial 3D scene geometry is generated, but computational complexity increases
Solution Approach 1:
The patent segments the 3D reconstruction process into distinct stages: depth map generation, TSDF fusion, and mesh extraction. By dividing the complex computational task into manageable segments and processing them sequentially with optimized parameters, the system achieves high 3D geometry accuracy while controlling computational complexity through staged processing.
Data Source
AI summary
Depth maps are generated based on a sequence of posed images captured by a camera, the depth maps are fused into a truncated signed distance function (TSDF), and an initial estimate of 3-dimensional (3D) scene geometry is generated by extracting a 3D mesh via the TSDF. 3D embeddings are estimated for each vertex in the 3D mesh by mapping each vertex to a multi-view consistent plane embedding space such that vertices on a same plane map to nearly a same place in the embedding space. The vertices are clustered into 3D plane instances based on respective 3D embeddings and geometry information defined by the 3D mesh to create a planar representation of the scene. A location of a virtual element in a virtual world of an augmented reality game is determined based on the planar representation.