Weakly Supervised Graph Matching for Semantic Scene Graph Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional digital image processing systems face inefficiencies, inflexibility, and inaccuracies in encoding and interpreting information from digital images due to reliance on heavily annotated data and limited domain applicability of scene graph generation models.
Innovation Solution
A weakly-supervised graph matching neural network is employed to align ungrounded label graphs and visual graphs using a first-order graph matching model optimized via contrastive learning, enabling efficient and flexible generation of semantic scene graphs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional scene graph generation models are used, then they can process digital images, but they require heavily annotated data which increases data verification time and reduces efficiency
Solution Approach 1:
The patent applies partial annotation by using only entity labels from image captions rather than full bounding box annotations. The graph matching algorithm partially aligns the label graph with the visual graph, requiring only entity-level annotations instead of complete object detection annotations, thereby reducing data verification time while maintaining scene graph generation capability
Solution Approach 2:
The patent introduces an ungrounded label graph as an intermediary representation between the image caption and the visual graph. This intermediary structure allows the system to work with minimal annotations (entity labels only) while still enabling accurate scene graph generation through graph matching, thus reducing data verification requirements
2Adaptability or versatility
If conventional scene graph generation models are used, then they can generate scene graphs, but they have limited domain applicability which reduces flexibility
Solution Approach 1:
The patent creates a universal graph matching framework that can be applied across different domains without domain-specific model adjustments. The method uses general-purpose graph matching algorithms that work with any image caption and visual graph combination, enabling the system to handle diverse domains (natural scenes, synthetic images, different object categories) with a single unified approach
Solution Approach 2:
The patent segments the scene graph generation problem into independent graph matching tasks that can be solved separately. By dividing the problem into entity alignment and relationship inference components, the system achieves domain versatility without increasing overall model complexity, as each segment uses simple, domain-agnostic operations
3Measurement precision
If heavily annotated data is used for training, then model accuracy improves, but computational resources and training time increase
Solution Approach 1:
The patent uses partial annotations (entity labels from captions only) instead of full annotations (bounding boxes, object IDs, relationships). This partial annotation approach maintains model accuracy by providing sufficient entity information while dramatically reducing the computational resources required for data preparation and model training
Solution Approach 2:
The system uses existing image captions as self-generated annotations, eliminating the need for separate manual annotation processes. The caption provides entity labels that directly serve the graph matching task, reducing computational overhead for data processing while maintaining accuracy through the use of pre-existing descriptive information
4Measurement precision
If exact graph matching is used to align label graphs and visual graphs, then alignment accuracy improves, but computational complexity increases
Solution Approach 1:
The patent segments the graph matching problem into entity-level alignment and relationship-level inference stages. The first stage matches entities between label graph and visual graph using simplified criteria, while the second stage infers relationships from aligned entities. This segmentation achieves accurate alignment without the computational complexity of exact graph matching
Solution Approach 2:
The patent uses partial graph matching by focusing only on entity alignment rather than complete graph isomorphism. The system performs sufficient alignment to identify corresponding entities and infer relationships, without requiring exact matching of all graph properties, thereby reducing computational complexity while maintaining practical accuracy
Data Source
AI summary
This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize weakly supervised graph matching to align an ungrounded label graph and a visual graph corresponding to a digital image. Specifically, the disclosed system utilizes a label embedding model to generate label graph embeddings from the ungrounded label graph and a visual embedding network to generate visual graph embeddings from the visual graph. Additionally, the disclosed system determines similarity metrics indicating the similarity of pairs of label graph embeddings and visual graph embeddings. The disclosed system then generates a semantic scene graph by utilizing a graph matching algorithm to align the ungrounded label graph and the visual graph based on the similarity metrics. In some embodiments, the disclosed system utilizes contrastive learning to modify the embedding models. Furthermore, in additional embodiments, the disclosed system utilizes the semantic scene graph to train a scene graph generation neural network.