Fine-Grained Action Classification Using Pose-Object Video Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for human action classification and regression lack the necessary granularity to assess compliance with specific procedural standards, particularly in scenarios requiring nuanced evaluations such as worker assembly actions or elderly motor skills, leading to inefficiencies and reliance on skilled personnel for quality assurance.
Innovation Solution
A method involving spatial-temporal video analysis that integrates pose and object datasets to generate a compound data structure, which is input into a machine learning model for fine-grained action classification and regression, enabling accurate assessment of human actions against specified standards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional hand-crafted features and rule-based systems are used for action classification, then the system is simpler to implement, but the measurement precision and ability to generalize across diverse scenarios deteriorates
Solution Approach 1:
The patent replaces traditional hand-crafted feature extraction and rule-based classification systems with deep learning-based automated feature learning and classification. The system uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to automatically learn hierarchical features from raw video data, eliminating the need for manual feature engineering and achieving superior classification precision across diverse action scenarios
Solution Approach 2:
The patent transforms the approach by changing from fixed hand-crafted features to dynamically learned features through deep learning models. The system adapts its feature representations through training on diverse action datasets, allowing it to automatically adjust to different action types, contexts, and variations without requiring manual reconfiguration of feature extraction parameters
2Measurement precision
If deep learning techniques are used for action classification, then the measurement precision and accuracy improve, but the computational resources and processing time increase
Solution Approach 1:
The patent segments the video processing task into multiple stages: first extracting spatial features from individual frames using CNNs, then processing temporal relationships using RNNs. This segmentation allows the system to handle complex action classification by breaking down the computation into manageable steps, processing only necessary frames and features at each stage, thereby reducing overall computational energy consumption while maintaining high accuracy
3Measurement precision
If current action classification methods are used, then the system can identify basic actions, but the ability to assess compliance with specific procedural standards and provide fine-grained evaluation deteriorates
Solution Approach 1:
The patent applies local quality analysis by examining specific spatial regions and temporal segments of the video data. The system identifies and analyzes particular action components, body parts, or object interactions that are relevant to compliance assessment, providing fine-grained evaluation of procedural standards. This localized analysis preserves critical compliance information that would otherwise be lost in coarse-grained action classification
Data Source
AI summary
Methods and a non-transitory computer-readable storage medium for fine-grained action classification and/or regression are disclosed. The method includes: receiving a video stream capturing a sequence of human subject actions; identifying reference objects with spatial-temporal relationships to the action sequence; extracting a pose dataset representing the action sequence; extracting object datasets representing spatial positions of the reference objects; generating a compound data structure integrating the pose dataset and object datasets; and inputting the compound data structure into a trained machine learning model for classification and/or regression.