Learnable Cost Volume for Accurate Pixel Correspondence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining pixel correspondence between images, such as those used in optical flow and stereo disparity calculations, face challenges in accounting for correlations among feature vector dimensions and distinguishing discriminative from noisy information, particularly in scenarios with large pixel displacements, textureless regions, and illumination changes.
Innovation Solution
The use of a learned elliptical inner product space defined by a symmetric and positive definite kernel matrix, which transforms feature vectors and weights their dimensions to improve similarity metric calculations, allowing for more accurate pixel correspondence determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a Euclidean inner product is used to compare feature vectors, then the calculation is simple and computationally efficient, but it fails to account for correlations among feature vector dimensions and cannot distinguish discriminative from noisy information
Solution Approach 1:
The kernel matrix is pre-computed and trained offline to capture correlations among feature vector dimensions. This preliminary action transforms the complex online computation into a simpler operation that only requires matrix multiplication with pre-stored weights, resolving the contradiction between accuracy and computational complexity
Solution Approach 2:
The kernel matrix acts as an intermediary that transforms the raw feature vectors into a space where correlations are accounted for and discriminative dimensions are emphasized. This intermediary structure enables accurate pixel correspondence determination without requiring complex real-time computations
2Measurement precision
If all feature vector dimensions are weighted equally in the similarity calculation, then the computation is straightforward, but noisy information cannot be distinguished from discriminative information
Solution Approach 1:
The kernel matrix assigns different weights to different dimensions of the feature vectors based on their discriminative quality. This local differentiation allows the system to emphasize important dimensions while suppressing noisy ones, resolving the contradiction between measurement precision and computational complexity
Solution Approach 2:
The kernel matrix transforms the uniform weighting parameter into dimension-specific weights that are learned from training data. This parameter change enables the system to adaptively weight features according to their discriminative power, improving similarity metric accuracy
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method includes obtaining a first plurality of feature vectors associated with a first image and a second plurality of feature vectors associated with a second image. The method also includes generating a plurality of transformed feature vectors by transforming each respective feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The method additionally includes generating a cost volume by determining, for each respective transformed feature vector of the plurality of transformed feature vectors, a plurality of inner products, wherein each respective inner product of the plurality of inner products is between the respective transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The method further includes determining, based on the cost volume, a pixel correspondence between the first image and the second image.