Scene Graph Training With Logic Constraints for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scene graph generation (SGG) methods rely heavily on neural networks, requiring large amounts of training data and struggle to leverage common sense knowledge efficiently, and existing symbolic logic approaches correct violations at inference-time, which is inefficient.
Innovation Solution
A method that calculates a loss function combining supervised and logic-based losses during training, using common-sense knowledge to update neural network weights, incorporating integrity constraints from knowledge bases like ConceptNet or ATOMIC to improve training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network techniques are used exclusively for scene graph generation, then the model can learn complex patterns, but huge amounts of training data are required and common sense knowledge cannot be efficiently leveraged
Solution Approach 1:
The patent combines neural network techniques with symbolic logic reasoning by integrating a logic-based loss function that incorporates common sense knowledge from knowledge bases. This hybrid approach merges the pattern recognition strength of neural networks with the logical reasoning capability of symbolic systems, allowing the model to achieve high accuracy without requiring huge amounts of training data.
Solution Approach 2:
The patent introduces integrity constraints from knowledge bases (such as ConceptNet or ATOMIC) as an intermediary between the neural network and the scene graph generation process. These constraints act as a mediator that guides the neural network's predictions by providing logical rules and common sense knowledge, reducing the reliance on large training datasets while improving generation accuracy.
2Reliability
If symbolic logic approaches are used to leverage common sense knowledge, then inference-time corrections can be made, but processing efficiency decreases due to post-processing requirements
Solution Approach 1:
The patent applies preliminary action by incorporating logical constraints and common sense knowledge into the training process itself, rather than correcting violations after inference. The logic-based loss function guides the neural network to learn logically consistent patterns during training, so that the model produces logically sound predictions directly without requiring post-processing corrections, thereby maintaining high inference speed.
Solution Approach 2:
The patent implements feedback mechanisms during training by using the logic-based loss function to provide continuous guidance to the neural network. The integrity constraints from knowledge bases serve as feedback signals that help the model learn to satisfy logical requirements, enabling the system to achieve both logical consistency and processing efficiency simultaneously.
3Productivity
If integrity constraints from knowledge bases are incorporated into training, then common sense knowledge is efficiently leveraged, but the complexity of the training process increases
Solution Approach 1:
The patent applies local quality by selectively incorporating integrity constraints from knowledge bases into the training process. Rather than applying all possible constraints uniformly, the system focuses on relevant constraints that are locally applicable to each scene graph generation task. This selective approach efficiently leverages common sense knowledge while avoiding unnecessary complexity from irrelevant constraints.
Data Source
AI summary
Broadly speaking, the disclosure generally relates to relates to a computer-implemented methods and systems for scene graph generation, and in particular for training a machine learning, ML, model to generate a scene graph. The method includes inputting training a training image into a machine learning model, outputting a predicted label for at least two objects in the training image and a predicted label for a relationship between the at least two objects. The training method includes calculating a loss, which takes into account both a supervised loss calculated by comparing the predicted labels to the actual labels for the training image, and a logic-based loss calculated by comparing the predicted labels to stored integrity constraints comprising common-sense knowledge. Advantageously, this means that the performance of the model is improved without increasing processing at inference-time.