Information Processing Apparatus for Visual Entity Relation Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scene graph generation techniques fail to accurately recognize the relation between visual entities in an image, as they miss contextually important information outside the bounding box regions of detected entities.
Innovation Solution
An information processing program that extracts a first feature from an image, detects visual entities, generates a second feature by combining detected visual entities, creates a relation map indicating the relation between visual entities, and estimates the relation using an attention map.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only bounding box regions of detected entities are used for scene graph generation, then the processing scope is limited and computationally simpler, but contextually important information outside these regions is missed, reducing relation recognition accuracy
Solution Approach 1:
The patent divides the image processing into two distinct segments: (1) processing within bounding box regions to extract entity features, and (2) processing outside bounding box regions to extract contextual features. This segmentation allows the system to handle different regions with different purposes, improving relation recognition while managing computational complexity through structured division of the processing area.
Solution Approach 2:
The patent applies different processing qualities to different regions of the image. Bounding box regions receive entity-focused processing to extract visual entity features, while regions outside bounding boxes receive context-focused processing to extract relational features. This local quality differentiation ensures that each region is processed with the appropriate level of detail and focus, improving overall relation recognition accuracy.
2Loss of information
If only entity features within bounding boxes are processed, then the feature extraction is more focused and efficient, but the contextual information necessary for accurate relation estimation is lost
Solution Approach 1:
The patent adds a new dimension to feature extraction by generating two distinct types of features: entity features from bounding box regions and contextual features from regions outside bounding boxes. This dimensional expansion in feature space allows the system to capture both entity characteristics and relational context simultaneously, preventing information loss while managing complexity through structured feature generation.
Solution Approach 2:
The patent introduces an attention mechanism as an intermediary component that selectively weights and integrates features from different regions. This attention mechanism acts as a mediator between the raw visual features and the final relation estimation, allowing the system to process contextual information efficiently by focusing computational resources on the most relevant features rather than processing all features uniformly.
3Measurement precision
If the system processes only detected visual entities, then the processing speed is faster and computational resources are conserved, but the relation estimation accuracy deteriorates due to missing contextual cues
Solution Approach 1:
The patent implements partial processing by selectively extracting features only from relevant regions rather than processing the entire image uniformly. By focusing computational effort on bounding box regions for entity features and only on necessary contextual regions for relational features, the system achieves improved relation estimation accuracy without the computational cost of processing the entire image at full resolution, thus maintaining processing efficiency.
Data Source
AI summary
A non-transitory computer-readable recording medium stores an information processing program for causing a computer to execute a process including: extracting a first feature from an image; detecting, from the extracted first feature, a plurality of visual entities included in the image; generating a second feature in which the visual entities in at least one combination of the plurality of detected visual entities are combined, in first feature, with each other; generating, based on the first feature and the second feature, a first map that indicates relation of each visual entity; extracting a fourth feature based on the first map and a third feature obtained by converting the first feature; and estimating the relation from the fourth feature.