PoTion Heatmap Aggregation for CNN Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for human action recognition using Convolutional Neural Networks (CNNs) face challenges in accurately recognizing actions due to the high variability and complexity of human actions, requiring extensive training time and careful initialization, and often struggle with recognizing certain actions efficiently.
Innovation Solution
A method for training CNNs that generates heatmaps for keypoint positions in videos, colorizes them based on relative time, aggregates these heatmaps into compact representations (PoTion) that combine appearance and motion, and uses these representations for training and classification, allowing for faster and more efficient action recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the I3D two-streams approach is used for human action recognition, then classification accuracy is improved, but training time increases to several days and careful initialization is required
Solution Approach 1:
The patent segments the video data processing into distinct functional streams: an appearance stream that processes RGB frames and a motion stream that processes optical flow data. Each stream is independently trained on different aspects of the video content, allowing parallel processing and faster convergence while maintaining comprehensive action recognition capability
Solution Approach 2:
The patent applies preliminary action by pre-processing video frames to extract optical flow data and generating motion representations before the main training process. This preparation of motion information in advance allows the network to focus on learning classification patterns rather than computing motion from scratch during training, reducing training time
2Measurement precision
If the I3D two-streams approach is used for human action recognition, then classification accuracy is improved, but device complexity increases due to multiple streams and careful initialization requirements
Solution Approach 1:
The patent merges the appearance and motion streams at the classification layer, combining their feature representations to produce the final action classification. This merging approach allows the system to leverage both appearance and motion information while maintaining separate, simpler processing paths that are easier to train and initialize compared to a fully integrated complex model
3Ease of manufacture
If traditional CNN training methods are used for action recognition, then training can be performed with standard approaches, but recognition accuracy remains insufficient for highly variable actions
Solution Approach 1:
The patent changes the parameter representation by transforming video data into optical flow fields and motion representations, converting spatial-temporal video information into a format that explicitly captures motion dynamics. This parameter transformation allows the CNN to more effectively learn action patterns while maintaining standard training procedures
Data Source
Figure 1
Figure 2(a0)~2(a2)
Figure 3a~3b
AI summary
The present invention relates to a method for training a CNN for classification of actions performed by subjects in videos; the method being characterized in that it comprises the implementation of steps of: (a) For each video of a base of training videos, (a0) for each frame (t ∈ 1; T ) of said video, for each of at least one keypoint (j ∈ 1; n ) of the subject, generating a heatmap Hjt of the keypoint (j) representing a position estimation of the keypoint (j) within said frame (t); (a1) colorizing each heatmap as a function of the relative time of the corresponding frame (t) in the video; (a2) for each keypoint (j ∈ 1; n ), aggregating all the colorized heatmaps Cjt of the keypoint (j) into at least one image (Sj, Uj, Ij, Nj) representing the evolution of the position estimation of the keypoint (j) during the video; (b) Training the CNN using as input the sets associated to each training video of images (Sj, Uj, Ij, Nj) representing the evolution of the position estimation of each keypoint (j) during the video. A method for classification of an action performed by a subject in an inputted video is further proposed.