Semantic Scene Graphs Using External Knowledge and Image Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scene graph generation techniques suffer from biases towards common object relationships due to limited datasets and noise contamination, leading to inaccurate scene graph representations.
Innovation Solution
The system generates scene graphs by using external knowledge to refine object and subgraph proposals, and performs image-level supervision through image reconstruction to regularize the scene graph generation model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional systems train on limited datasets with long-tail distributions, then training is feasible with available data, but the systems become biased towards the most common relationships resulting in misidentified object relationships
Solution Approach 1:
The system pre-trains object detection models on large-scale pre-trained models before fine-tuning on task-specific datasets. This preliminary action allows the model to learn general object detection capabilities and relationship patterns from diverse data sources before specializing, thereby reducing bias towards common relationships and improving detection accuracy for rare object relationships.
Solution Approach 2:
The system introduces an external knowledge base as an intermediary to guide and correct relationship predictions. The knowledge base provides prior knowledge about object relationships that helps the model distinguish between common and rare relationships, reducing the bias towards frequent relationships and improving overall relationship detection accuracy.
2Adaptability or versatility
If large scale datasets with crowd-sourced information are used for training, then more relationship triplets are available, but noise contamination with missing annotations and meaningless proposals results in poor performance
Solution Approach 1:
The system extracts and removes noisy or meaningless proposals from the training data through filtering mechanisms. By taking out low-quality annotations and irrelevant proposals from the large-scale crowd-sourced datasets, the system retains only the valuable relationship triplets for training, thereby maintaining data diversity while improving prediction accuracy.
Solution Approach 2:
The system implements feedback mechanisms where relationship predictions are continuously refined based on consistency checks with the external knowledge base and cross-validation with multiple detection models. This feedback loop allows the system to correct errors from noisy annotations and improve predicate label accuracy iteratively.
3Ease of operation
If conventional systems rely on object detection performance for predicate inference, then object identification is straightforward, but poor object detection performance reduces predicate inference accuracy
Solution Approach 1:
The system introduces an external knowledge base as an intermediary between object detection and predicate inference. This knowledge base provides prior knowledge about plausible relationships between objects, allowing the system to correct errors from poor object detection and improve predicate inference accuracy even when detection performance is limited.
Solution Approach 2:
The system changes the parameters of the inference process by incorporating confidence thresholds and multiple hypothesis testing. Instead of directly inferring predicates from object detection results, the system adjusts inference parameters to account for detection uncertainty, thereby improving the reliability of scene graph representations.
Data Source
AI summary
Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.