Person Intention Reasoning via Occluded Joint Feature Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms for visual commonsense reasoning (VCR) tasks suffer from low accuracy due to the coarse granularity of human features extracted from target detection networks trained on datasets like Visual Genome or COCO.
Innovation Solution
A person intention reasoning method that performs target detection, determines detection bounding boxes for persons, acquires joint features and occlusion probabilities, predicts and corrects features using an occluded joint prediction network, and finally performs person intention reasoning using the corrected features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If target detection networks trained on Visual Genome or COCO are used for visual feature extraction, then the device complexity is reduced and ease of operation is improved, but the granularity of human features becomes coarse leading to lower reasoning accuracy
Solution Approach 1:
The patent segments the human body into multiple joints (e.g., 18 joints including head, shoulders, elbows, wrists, hips, knees, ankles) and extracts features for each joint individually. This segmentation transforms coarse-grained whole-person detection into fine-grained joint-level feature extraction, directly improving reasoning accuracy while maintaining manageable system complexity through modular processing of each joint's bounding box and features.
2Measurement precision
If existing target detection methods are used, then the ease of manufacture is improved, but feature missing for occluded person parts occurs leading to lower accuracy
Solution Approach 1:
The patent performs preliminary occlusion prediction by training a prediction network on fused features (joint features combined with occlusion probabilities) to predict which joints may be occluded before final reasoning. This preliminary action allows the system to proactively identify and handle occluded joints, ensuring feature completeness without requiring complex post-processing or re-detection mechanisms.
Solution Approach 2:
The patent implements feedback by using occlusion probabilities as a feedback signal to guide the feature extraction and reasoning process. The occlusion prediction network provides feedback about which joints are likely occluded, allowing the system to adjust its feature extraction strategy, prioritize visible joints, or apply correction mechanisms, thereby improving feature completeness while maintaining implementation simplicity through a closed-loop approach.
3Measurement precision
If coarse-grained human features are extracted from target detection, then the device complexity is reduced, but the person intention reasoning accuracy deteriorates
Solution Approach 1:
The patent transitions from two-dimensional whole-person bounding box detection to a multi-dimensional joint-level feature space by extracting coordinates and features for 18 different joints. This dimensional expansion from coarse whole-person features to fine-grained joint features significantly improves intention reasoning accuracy. The increased complexity is managed through efficient coordinate processing and fused feature representation that leverages the structured nature of joint data.
Data Source
AI summary
The person intention reasoning method includes: performing object detection on a to-be-reasoned image to obtain an object detection result; determining that an image portion corresponding to a detection bounding box of each person in the to-be-reasoned image is a to-be-reasoned sub-image of the corresponding person respectively, and acquiring a joint feature and an occlusion probability of a joint of the corresponding person; performing prediction and analysis on the joint feature of corresponding joint based on the occlusion probability to obtain a corresponding prediction feature, and performing correction based on the joint feature and the prediction feature of the joint of the corresponding person to obtain a corresponding correction feature; and performing person intention reasoning by using the object detection result and the correction feature of the joint of the corresponding person to obtain a corresponding person intention reasoning result.