Patch-Based Pixel Correspondence With Local Neighborhood Consensus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for computing dense semantic correspondences between images with large intraclass appearance and shape variations are computationally expensive and inefficient, often requiring high memory usage and slow computation due to exhaustive matching and 4D convolutions.
Innovation Solution
The PatchMatch Neighborhood Consensus (PMNC) method uses a convolutional neural network-based feature extractor and a neighborhood consensus-based scoring function to refine pixel correspondences by applying 4D convolutions locally on selective patches, reducing computational complexity and memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exhaustive matching and 4D convolutions are used to compute dense semantic correspondences, then matching accuracy is improved, but computational complexity and memory usage increase significantly
Solution Approach 1:
The patent divides the feature maps into multiple patches and processes them separately using parallel computation. Instead of performing exhaustive matching on the entire feature maps, the method segments them into smaller regions that can be processed independently, reducing computational complexity while maintaining matching accuracy through localized patch-based operations
Solution Approach 2:
The patent transforms the 4D convolution operation into a 2D convolution operation by reorganizing the computational dimensions. This dimensionality reduction converts the computationally expensive 4D convolution into a more efficient 2D convolution, significantly reducing computational complexity and memory usage while preserving the essential matching information
2Measurement precision
If exhaustive matching is performed on entire feature maps, then correspondence accuracy is improved, but computation time increases
Solution Approach 1:
The patent segments the feature maps into multiple patches and processes them in parallel. This segmentation allows the computation to be distributed across multiple processing units simultaneously, reducing the overall computation time while maintaining correspondence accuracy through the parallel evaluation of multiple patch regions
Solution Approach 2:
The patent performs matching on a selective subset of patches rather than exhaustively processing the entire feature maps. By identifying and processing only the most relevant patches that contain meaningful correspondence information, the method reduces computation time while maintaining sufficient accuracy for the application
3Measurement precision
If 4D convolutions are applied to the entire search space, then scoring accuracy is improved, but memory footprint increases
Solution Approach 1:
The patent divides the search space into multiple patches and processes them separately. This segmentation reduces the memory footprint by loading and processing only one patch at a time rather than requiring the entire search space to be held in memory simultaneously, while maintaining scoring accuracy through systematic evaluation of all patches
Solution Approach 2:
The patent extracts and processes only the necessary patch regions from the full search space rather than processing the entire space. By extracting and evaluating selective patches that contain the most relevant correspondence information, the method reduces memory footprint while preserving scoring accuracy for the critical regions
Data Source
Figure 1
Figure 2
Figure 3
AI summary
One example provides a computing system comprising a storage machine storing instructions executable by a logic machine to extract features from a source and target images to form source and target feature maps, form a correlation map comprising a plurality of similarity scores, form an initial correspondence map comprising initial mappings between pixels of the source feature map and corresponding pixels of the target feature map, refine the initial correspondence map by, for each of one or more pixels of the source feature map, for each of a plurality of candidate correspondences, inputting a four-dimensional patch into a trained scoring function, the trained scoring function being configured to output a correctness score, and selecting a refined correspondence based at least upon the correctness scores, and output a refined correspondence map comprising a refined correspondence for each of the one or more pixels of the source feature map.