Visual Scene Graph Model Training with Common Sense Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual scene graph models face challenges in performing equally well across all categories due to the long-tail distribution issue, where rare relationships are underrepresented in training data, leading to overfitting on frequent relationships and poor performance on rare cases.
Innovation Solution
A method that combines datasets containing spatial and causal/intention relationships to generate learned common sense knowledge embeddings, which are then used to train a visual scene graph model, addressing the long-tail bias by incorporating diverse and nuanced relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If training data is collected from real-world scenarios, then common relationships are well-represented, but rare relationships are underrepresented leading to long-tail bias
Solution Approach 1:
The patent applies preliminary action by pre-training the visual scene graph model on a large-scale common sense knowledge graph before fine-tuning on domain-specific data. This pre-training establishes a foundational understanding of rare relationships and commonsense knowledge that compensates for their underrepresentation in the training data, allowing the model to generalize better to rare cases without requiring extensive rare relationship examples
Solution Approach 2:
The patent introduces a commonsense knowledge graph as an intermediary resource that bridges the gap between common and rare relationships. This knowledge graph serves as a mediator that provides additional contextual information and relationship patterns that help the model understand and predict rare relationships, effectively transferring knowledge from common to rare cases through the intermediary knowledge base
2Measurement precision
If the model focuses on frequent relationships during training, then performance on common cases improves, but performance on rare cases deteriorates
Solution Approach 1:
The patent applies local quality by implementing different training strategies for different relationship types. The model uses focal loss weighting that applies different loss weights to different relationship categories, allowing aggressive optimization on common relationships while maintaining sufficient attention on rare relationships. This localized differential treatment enables the model to achieve high accuracy on common cases without completely neglecting rare cases
Solution Approach 2:
The patent changes parameters dynamically during training by adjusting the loss function weights and sampling probabilities based on relationship frequency. The training process modifies these parameters to balance the contribution of different relationship types, preventing the model from completely overfitting to frequent relationships while still achieving high precision on common cases through optimized parameter settings
3Measurement precision
If manual annotation is performed to create ground-truth data, then data quality improves, but annotation bias toward obvious relationships increases
Solution Approach 1:
The patent applies feedback by using the trained model to generate predictions that are then compared against ground-truth annotations. The discrepancy between model predictions and annotations provides feedback that highlights potentially missed rare relationships. This feedback loop allows the system to iteratively improve by identifying and correcting annotation biases, gradually incorporating previously overlooked rare relationships into the training data
Solution Approach 2:
The patent enables self-service by allowing the model to autonomously identify and flag potential annotation biases through its predictions. The system automatically generates candidate relationships that may have been missed by annotators and prioritizes these for review, reducing reliance on manual inspection and enabling the system to self-correct annotation biases without extensive human intervention
Data Source
AI summary
A method for controlling a device to manage a visual scene graph model can include obtaining, via a processor in the device, a first dataset; obtaining, via the processor, a second data set different from the first dataset, the second dataset including one or more of a causal relation or an intention relation; combining, via the processor, the first dataset and the second dataset to generate a combined dataset. Also, the method can further include applying a knowledge embedding function to the combined dataset to generate learned common sense knowledge embeddings; and training a visual scene graph model based on the learned common sense knowledge embeddings to generate a trained visual scene graph model. Further, the method can include executing a function based on an output of the trained visual scene graph model. The device can include at least one of a smart television, a mobile phone or a robot.