Adaptive Clustering Learning for Visual Relationship Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing visual relationship detection methods ignore latent information between different visual relationships and struggle with complex relationship modeling, leading to inadequate performance in large-scale visual relationship detection.

Innovation Solution

A visual relationship detection method based on adaptive clustering learning, which involves detecting visual objects, embedding context representations into low-dimensional subspaces, and using clustering-driven attention mechanisms to regularize and fuse representations for improved predicate prediction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If visual relationships are modeled in a unified joint subspace, then the detection process is simplified, but latent relatedness information between different visual relationships is ignored

Engineering Contradiction:

Improvemodeling complexityVSAvoidlatent relatedness information

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent segments the unified joint subspace into multiple clustering subspaces, where each subspace captures specific types of visual relationship patterns. This segmentation allows the model to preserve latent relatedness information between different visual relationships while maintaining manageable complexity through organized decomposition.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension by creating multiple clustering subspaces from the original joint subspace. This dimensional transformation enables the model to capture latent relatedness information that would be lost in a single unified subspace, while the subspaces can be processed independently to control complexity.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If visual relationship detection separates object detection and predicate detection into branches, then scalability improves, but relationship modeling becomes more complex

Engineering Contradiction:

ImprovescalabilityVSAvoidrelationship modeling complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces context representations as an intermediary between object detection and predicate detection branches. These context representations capture spatial and semantic information, serving as a bridge that simplifies relationship modeling while maintaining the scalability benefits of separated detection branches.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent combines context representations with object features to create composite representations for relationship prediction. This composite approach integrates information from multiple sources in a structured manner, managing modeling complexity while preserving the scalability of the separated-branch architecture.

Inventive Principle:

Principle #40Composite materials

3Ease of operation

If all visual relationships are identified in a unified joint subspace, then processing is simpler, but fine-grained recognition of visual relationship subclasses is inadequate

Engineering Contradiction:

Improveprocessing simplicityVSAvoidfine-grained recognition accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent segments the unified joint subspace into multiple clustering subspaces, where each subspace is specialized for recognizing specific types of visual relationships. This segmentation maintains processing simplicity through organized structure while enabling fine-grained recognition by dedicating specific subspaces to specific relationship patterns.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by making each clustering subspace specialized for specific types of visual relationships rather than treating all relationships uniformly. This allows each subspace to develop specialized features and representations optimized for its specific relationship type, improving fine-grained recognition accuracy while maintaining overall processing efficiency.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11361186B2Visual relationship detection method and system based on adaptive clustering learning

Publication Date: 2022.06.14 TIANJIN UNIV

AI summary

The present disclosure discloses a visual relationship detection method based on adaptive clustering learning, including: detecting visual objects from an input image and recognizing the visual objects to obtain context representation; embedding the context representation of pair-wise visual objects into a low-dimensional joint subspace to obtain a visual relationship sharing representation; embedding the context representation into a plurality of low-dimensional clustering subspaces, respectively, to obtain a plurality of preliminary visual relationship enhancing representation; and then performing regularization by clustering-driven attention mechanism; fusing the visual relationship sharing representations and regularized visual relationship enhancing representations with a prior distribution over the category label of visual relationship predicate, to predict visual relationship predicates by synthetic relational reasoning. The method is capable of fine-grained recognizing visual relationships of different subclasses by mining latent relationships in-between, which improves the accuracy of visual relationship detection.