Real-Time Stereo Matching via Hierarchical Iterative Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning networks for stereo matching are computationally expensive and result in high latency, making them unsuitable for real-time applications such as mobile robots and self-driving vehicles, where fast and accurate depth map generation is required.
Innovation Solution
A real-time active stereo system that includes a capture system and a depth sensing computing system, utilizing a feature extractor to extract features at multiple resolutions, an initialization engine to generate initial depth estimations, and a propagation engine to iteratively refine these estimations through image warping and spatial propagation, allowing for efficient and accurate depth map generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large convolutional neural networks are used to improve depth map accuracy, then measurement precision is improved, but processing time increases causing high latency
Solution Approach 1:
The patent divides the stereo matching process into multiple passes operating at different resolutions. The first pass processes a downsampled version of the input images to generate an initial depth map, while subsequent passes refine this result at higher resolutions. This segmentation allows the system to capture global depth structures efficiently while progressively adding fine-grained details, thereby achieving high accuracy without processing the full-resolution images in a single computationally expensive pass.
Solution Approach 2:
The patent performs preliminary depth estimation on downsampled images before processing the full-resolution images. By generating an initial depth map from lower-resolution inputs first, the system establishes a rough depth structure that guides subsequent refinement steps. This preliminary action reduces the search space for later high-resolution processing, enabling faster convergence to accurate depth maps while minimizing overall processing time.
2Measurement precision
If computationally expensive deep learning networks are used, then depth map accuracy is improved, but productivity decreases due to high latency
Solution Approach 1:
The patent segments the depth estimation task into multiple resolution levels, processing images at progressively higher resolutions across different passes. This approach breaks down the computationally intensive full-resolution processing into manageable stages, where each pass contributes to the final accuracy while maintaining acceptable processing speeds for real-time applications.
Solution Approach 2:
The patent implements a multi-pass refinement approach where subsequent passes progressively improve depth map quality by incorporating details from higher resolutions. The system performs partial processing at each resolution level rather than attempting to process full-resolution images in a single pass, achieving near-complete accuracy through cumulative refinement while maintaining real-time performance through efficient resource utilization at each stage.
Data Source
AI summary
According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.