Skeleton-Image Fusion for Interactive Behavior Understanding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing human behavior understanding methods that rely solely on skeleton information struggle with accurately interpreting interactive behaviors, such as 'human-object' and 'human-human' interactions, due to the loss of environmental context and noise interference in image data.
Innovation Solution
An interactive behavior understanding method that combines skeleton features extracted using a graph convolution network with image features obtained through a Vision Transformer network and multi-head attention mechanism, allowing for the retention of effective image information and enhanced environmental perception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If skeleton information is used for behavior understanding, then noise interference is reduced and pure behavior information is obtained, but effective image information such as environment is lost
Solution Approach 1:
The patent segments the input data into two distinct streams: skeleton information stream and image information stream. The skeleton stream processes pose data through GCN and attention mechanisms to extract behavioral features, while the image stream processes visual data through Vision Transformer to extract environmental context features. This segmentation allows each stream to be optimized independently, reducing noise in skeleton processing while preserving effective information in image processing.
Solution Approach 2:
The patent creates a composite feature representation by fusing skeleton features and image features at multiple levels. The fusion mechanism combines the pure behavioral information from skeleton data with the environmental context from image data, creating a comprehensive feature vector that leverages the strengths of both modalities while compensating for their individual limitations.
2Quantity of substance
If image information is used for behavior understanding, then enough image features and human body features are obtained, but noise interference information increases which is not conducive to behavior understanding
Solution Approach 1:
The patent extracts relevant features from the image data while discarding noise through the Vision Transformer architecture. The multi-head attention mechanism selectively attends to important regions and features in the image, extracting meaningful environmental context and human body features while filtering out redundant noise information. This extraction process obtains necessary image features without carrying over harmful noise.
3Device complexity
If simple skeleton data is used for interactive behavior understanding, then processing is simplified, but accuracy in distinguishing human-object and human-human interactions is poor
Solution Approach 1:
The patent introduces dynamic processing mechanisms including Bahdanau attention for skeleton data and multi-head attention for image data. These dynamic mechanisms adaptively weight different features based on their relevance to the current interaction context, allowing the system to focus on critical information for distinguishing human-object and human-human interactions while maintaining efficient processing through learned attention patterns.
Solution Approach 2:
The patent introduces an intermediate fusion layer that mediates between simple skeleton data and complex interaction recognition. This intermediate representation combines skeleton features with corresponding image features, creating a bridge that enriches the simple skeleton data with environmental context without directly exposing the full complexity of image processing to the behavior classification task.
Data Source
AI summary
The present invention discloses an interactive behavior understanding method for posture reconstruction based on skeleton and image features. The steps are as follows: constructing and preprocessing the data set, extracting skeleton and image features, fusing and reconstructing these features, and conducting experimental evaluation and validation. This method retains the purity of skeleton features for human behavior information extraction and uses image features to retain effective environmental information, complementing the model feature information. Skeleton features are extracted using a graph convolution network, enhancing the relevance of input skeleton point information for accurate feature extraction. Effective image features are quickly and accurately extracted through the Vision Transformer network combined with a multi-head attention mechanism.