Compositional HOI Detection for Rare Human-Object Interactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing human activity recognition systems face data sparsity issues, particularly in detecting human-object interactions (HOIs) due to the difficulty in obtaining comprehensive training datasets that cover all possible HOIs and objects, leading to inaccurate or failed detections of rare interactions.
Innovation Solution
A compositional model is employed that integrates language and visual models to process semantic and visual embeddings, determining scores based on cosine distances to identify HOIs, allowing for the detection of interactions not present in the training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comprehensive training datasets covering all possible HOIs and objects are obtained, then detection accuracy improves, but data collection difficulty and time increase
Solution Approach 1:
The system segments HOI detection into three independent embedding components: subject embeddings, verb embeddings, and object embeddings. This segmentation allows the model to process and reuse individual components rather than requiring complete training examples for every possible HOI combination, thereby improving detection accuracy without proportionally increasing data collection requirements.
Solution Approach 2:
The embedding models are designed to be universal and reusable across different HOI scenarios. The subject, verb, and object embeddings can be combined in various ways to detect different interactions, allowing the system to generalize from limited training data to recognize rare or unseen HOIs without requiring extensive additional training data for each specific interaction type.
2Measurement precision
If comprehensive training datasets covering all possible HOIs and objects are obtained, then detection accuracy improves, but system complexity increases
Solution Approach 1:
By dividing the HOI detection task into separate embedding models for subjects, verbs, and objects, the system reduces the complexity of training and processing. Each embedding model can be trained independently on its respective component, avoiding the need to train a single complex model on all possible HOI combinations, thus managing system complexity while maintaining detection accuracy.
Solution Approach 2:
The embedding models serve as intermediaries that translate visual features into semantic representations. These embeddings act as a bridge between the input image and the final HOI classification, simplifying the overall system architecture by introducing a standardized intermediate representation layer that can be reused across different detection tasks.
3Measurement precision
If rare interactions are detected accurately, then detection accuracy improves, but data sparsity worsens
Solution Approach 1:
The segmentation of HOI into reusable subject, verb, and object embeddings allows the system to detect rare interactions by combining embeddings from individually trained models. This approach enables the system to generalize from limited training examples by recombining known components in novel ways, improving detection accuracy for rare HOIs without requiring proportional increases in training data quantity.
Solution Approach 2:
The system performs preliminary training of embedding models on available data before deployment. By pre-training the subject, verb, and object embeddings on the training dataset, the system prepares reusable components that can be combined to detect rare interactions during inference, eliminating the need to collect additional training data for each rare interaction type.
Data Source
AI summary
Implementations include actions of receiving an image; extracting a visual HOI and a set of visual embeddings, the visual HOI indicating a subject and an object; obtaining, using a vector library, a set of semantic HOIs and sets of semantic embeddings based on the subject, the object and a set of verbs included in the vector library, each set of semantic embeddings corresponding to a semantic HOI; processing, by a compositional model, the set of visual embeddings to provide a set of transition visual embeddings; processing the sets of semantic embeddings to provide respective sets of transition semantic embeddings; determining a set of scores based on the set of transition visual embeddings and the sets of transition semantic embeddings, each score representing a degree of similarity between the visual HOI and a semantic HOI; and determining at least one predicted HOI represented within the image based on the scores.