Phrase Grounding Model Contextual Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional noun-matching applications for phrase grounding disregard interrelationships between noun phrases, leading to inaccurate mappings and failure to match noun phrases with multiple regions in an image, resulting in inconsistent and incomplete results.
Innovation Solution
A computer-implemented method that generates sequential grounding decisions for each phrase in a source sentence based on other phrases and previously generated decisions, using a phrase grounding model that includes a training subsystem, pre-training engine, supervised training engine, and decision engine with recurrent neural networks to consider contextual relationships and generate multiple matches for each phrase.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional noun-matching applications independently map each noun phrase to at most one bounding box, then the processing complexity is reduced and operation speed is improved, but the mapping accuracy deteriorates and interrelationships between noun phrases are disregarded
Solution Approach 1:
The system dynamically adjusts the matching process by first performing independent noun phrase to bounding box mapping, then iteratively refining results by considering interrelationships between noun phrases. The decision engine dynamically updates matched pairs based on contextual information from other noun phrases, transforming a static independent mapping process into a dynamic contextual refinement process that improves accuracy while maintaining reasonable processing efficiency
Solution Approach 2:
The system implements feedback mechanisms where the mapping of one noun phrase influences the mapping of subsequent noun phrases. The decision engine uses previously generated matched pairs as feedback to inform subsequent matching decisions, allowing the system to correct early mistakes and ensure consistent interpretations of interrelationships between noun phrases throughout the processing sequence
2Device complexity
If conventional noun-matching applications map each noun phrase to at most one bounding box, then the device complexity is reduced, but the comprehensiveness of phrase grounding deteriorates when noun phrases actually match multiple regions
Solution Approach 1:
The system segments the matching process into distinct phases: initial independent mapping phase that generates candidate matched pairs, and subsequent refinement phase that considers interrelationships. This segmentation allows the system to handle multiple matches per noun phrase by generating multiple candidate pairs initially, then filtering and refining them based on contextual consistency without requiring a completely complex unified approach
Solution Approach 2:
The system performs partial matching by initially mapping each noun phrase to multiple potential bounding boxes rather than forcing a single match, then performs excessive action by generating more candidate matched pairs than ultimately needed. The decision engine subsequently filters these excessive candidates to retain only those consistent with interrelationships, allowing comprehensive matching capability while managing complexity through selective retention rather than preventing multiple matches from the start
Data Source
AI summary
In various embodiments, a phrase grounding model automatically performs phrase grounding for a source sentence and a source image. The phrase grounding model determines that a first phrase included in the source sentence matches a first region of the source image based on the first phrase and at least a second phrase included in the source sentence. The phrase grounding model then generates a matched pair that specifies the first phrase and the first region. Subsequently, one or more annotation operations are performed on the source image based on the matched pair. Advantageously, the accuracy of the phrase grounding model is increased relative to prior art solutions where the interrelationships between phrases are typically disregarded.