Semantic Segmentation for AR Depth Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality driving systems face challenges in accurately estimating the 3D position and orientation of vehicles using georeferenced coordinate systems and interpreting dynamic 3D scenes from camera images, with existing methods being costly, limited in range, or lacking in real-time capabilities.
Innovation Solution
The system employs a monocular video camera with a pretrained deep learning network to register 2D video frames with 3D reference data, using semantic segmentation to infer dynamic occlusion and recover absolute depth, while integrating GPS, IMU, and compass sensors for navigation, and generating high-resolution depth maps at 10 Hz frame rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LIDAR sensors and GPS devices are used to estimate 3D position and orientation, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent replaces complex mechanical sensing systems (LIDAR, GPS) with a computational approach using monocular video cameras and deep learning networks. The system uses semantic segmentation of video frames combined with pre-trained neural networks to infer 3D position and orientation, substituting expensive hardware with software-based solutions that achieve comparable measurement precision.
Solution Approach 2:
The system creates a virtual 3D model of the environment by processing 2D video frames through deep learning networks. Instead of directly measuring 3D space with LIDAR, the system copies and reconstructs spatial information from 2D image data, enabling pose estimation without expensive 3D sensing hardware.
2Measurement precision
If deep learning networks are used for semantic segmentation, then measurement precision is improved, but use of energy increases
Solution Approach 1:
The system uses pre-trained deep learning networks that have already learned semantic segmentation features during offline training. During runtime, the pre-trained models perform inference with reduced computational overhead compared to training from scratch, lowering energy consumption while maintaining high measurement precision for depth estimation and navigation.
3Productivity
If real-time processing at 10 Hz frame rates is implemented, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent replaces complex real-time 3D processing systems with optimized 2D video frame processing using deep learning. By working directly with 2D images and using pre-trained networks, the system achieves 10 Hz real-time processing rates with reduced computational complexity compared to traditional 3D point cloud processing or LIDAR data handling.
Data Source
AI summary
Techniques for augmenting a reality captured by an image capture device are disclosed. In one example, a system includes an image capture device that generates a two-dimensional frame at a local pose. The system further includes a computation engine executing on one or more processors that queries, based on an estimated pose prior, a reference database of three-dimensional mapping information to obtain an estimated view of the three-dimensional mapping information at the estimated pose prior. The computation engine processes the estimated view at the estimated pose prior to generate semantically segmented sub-views of the estimated view. The computation engine correlates, based on at least one of the semantically segmented sub-views of the estimated view, the estimated view to the two-dimensional frame. Based on the correlation, the computation engine generates and outputs data for augmenting a reality represented in at least one frame captured by the image capture device.