Semantic Computation Graphs for Referring Expression Grounding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional natural language processing systems struggle to accurately parse referring expressions, leading to misidentification of named entities and incorrect computation graphs for image editing, search, and captioning tasks due to the inability to capture all attributes, modifiers, and interrelationships between objects.
Innovation Solution
The system employs a shallow semantic parser and grounding ontology to generate a hierarchical (subject, predicate, object) structure, transforming it into a semantic computation graph that identifies all possible image objects, their attributes, and interrelationships, using lexicon ontologies to ensure consistent semantic mapping between language objects and computer vision labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional natural language processing systems are used to parse referring expressions, then the processing speed is maintained, but the accuracy of named entity identification and computation graph generation deteriorates
Solution Approach 1:
The patent introduces an intermediary semantic parsing layer that transforms natural language referring expressions into structured computation graphs. This intermediary representation captures semantic relationships between objects, attributes, and modifiers, serving as a bridge between raw text and downstream vision-language tasks, thereby improving accuracy without requiring direct modification of the base NLP system
Solution Approach 2:
The parsing system segments the referring expression into distinct computational components: object identification, attribute extraction, and relationship modeling. By dividing the complex parsing task into these manageable segments, the system achieves higher accuracy in named entity identification while maintaining organizational clarity and reducing overall system complexity
2Measurement precision
If a detailed semantic computation graph is generated to capture all attributes and interrelationships, then the accuracy of image editing and searching improves, but the computation time and graph complexity increase
Solution Approach 1:
The system performs preliminary semantic parsing and computation graph generation before the actual image editing or searching operation. By pre-processing the referring expression into a structured computation graph that captures all semantic relationships, the system prepares the data in advance, enabling faster and more accurate execution of downstream tasks without repeated parsing overhead
Solution Approach 2:
The computation graph is designed to be dynamic and adaptive, allowing the system to traverse only the relevant portions of the graph based on the specific task requirements. This dynamic traversal approach enables the system to capture detailed semantic relationships when needed while optimizing computation time by avoiding unnecessary graph processing for simpler queries
3Reliability
If lexicon ontologies are used to ensure consistent semantic mapping, then the reliability of language-to-vision label mapping improves, but the system complexity and processing overhead increase
Solution Approach 1:
The patent employs universal lexicon ontologies that serve multiple functions: they provide consistent semantic mapping between language objects and computer vision labels, enable generalization across different referring expressions, and support various downstream tasks including image editing, searching, and captioning. This multi-functionality achieves reliable semantic mapping without requiring separate specialized systems for each task
Solution Approach 2:
The system utilizes lexicon ontologies to standardize and normalize semantic parameters across different language expressions. By mapping diverse linguistic descriptions to standardized ontology parameters, the system achieves consistent and reliable language-to-vision label mapping, reducing variability and improving overall system reliability
Data Source
AI summary
Embodiments of the present invention provide systems, methods, and non-transitory computer storage media for parsing a given input referring expression into a parse structure and generating a semantic computation graph to identify semantic relationships among and between objects. At a high level, when embodiments of the preset invention receive a referring expression, a parse tree is created and mapped into a hierarchical subject, predicate, object graph structure that labeled noun objects in the referring expression, the attributes of the labeled noun objects, and predicate relationships (e.g., verb actions or spatial propositions) between the labeled objects. Embodiments of the present invention then transform the subject, predicate, object graph structure into a semantic computation graph that may be recursively traversed and interpreted to determine how noun objects, their attributes and modifiers, and interrelationships are provided to downstream image editing, searching, or caption indexing tasks.