RAW Stereo Depth Estimation Using CFA Color Channels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stereo vision methods for depth estimation using RGB images face challenges such as ambiguity in correspondence due to occlusion, illumination changes, and textureless regions, and rely on complex image signal processing (ISP) that introduces errors and computational overhead.
Innovation Solution
Perform depth estimation directly from RAW images using an end-to-end trainable neural network that processes color filter array (CFA) data without ISP, estimating disparity from color channels using separate or fused cost volumes and machine learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard ISP pipeline is used to process RAW images before depth estimation, then image quality is improved, but processing time and computational complexity increase significantly
Solution Approach 1:
The patent extracts and removes the ISP processing steps from the traditional pipeline, applying depth estimation directly to RAW images. This eliminates the time-consuming demosaicing, white balance, and tone mapping operations while maintaining depth estimation capability through direct processing of the raw sensor data.
Solution Approach 2:
The patent performs depth estimation as a preliminary action before ISP processing. By estimating depth from RAW images first, the system obtains depth information earlier in the pipeline, avoiding the need to wait for the complete ISP processing to finish, thus reducing overall processing time.
2Reliability
If ISP pipeline with demosaicing is used, then complete color information is obtained, but interpolation artifacts and errors are introduced
Solution Approach 1:
The patent removes the demosaicing step from the processing pipeline. Instead of interpolating missing color information through ISP, the system performs depth estimation directly on the RAW data with the color filter array pattern, avoiding the introduction of interpolation artifacts and errors entirely.
Solution Approach 2:
The patent inverts the traditional approach by not converting RAW to RGB through demosaicing. Instead, it processes the RAW data in its original form with the CFA pattern, using the existing color channel information directly for depth estimation without introducing interpolation errors.
3Measurement precision
If multiple color channels are processed separately, then depth estimation accuracy improves, but computational load increases
Solution Approach 1:
The patent merges the processing of multiple color channels by performing depth estimation operations that utilize information from all color channels simultaneously. The neural network processes the multi-channel RAW data in an integrated manner, combining the computational tasks rather than processing each channel separately, thus reducing overall complexity while maintaining accuracy.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
An device for performing depth estimation on first and second images captured using one or more cameras having a colour filter array, each image comprising multiple colour channels which each correspond to a respective colour channel of the colour filter array, the device being configured to perform depth estimation by estimating disparity from the colour channels of the images.