Human Coactivity Recognition With Inter-Person Keypoint Distance Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision approaches are limited in classifying human coactivities involving interactions between two or more people, as they focus on single-person activities and cannot effectively utilize the interaction information necessary for identifying coactivities like partner dance or soccer foul play.
Innovation Solution
A method using 2D or 3D distance measurements between key points on multiple human bodies to generate an encoded representation, which is then classified by a fully-connected neural network trained to identify coactivities in images or video frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D convolution-based neural networks are used for human action recognition, then classification accuracy for single-person activities is improved, but the system becomes incapable of classifying coactivities involving interactions between multiple people
Solution Approach 1:
The system segments the human bodies in the image into multiple key points (e.g., 18 key points per person). For each person, separate key point position sets are extracted and processed independently to generate encoded representations. This segmentation allows the system to handle multiple people while maintaining the ability to capture interaction information between them through subsequent distance calculations.
Solution Approach 2:
The system introduces an intermediary encoding process that calculates distances between key points of different people. The encoded representation includes not only intra-person key point distances but also inter-person distances, serving as a mediator that captures interaction information. This intermediary representation is then fed to the classifier, enabling coactivity recognition without modifying the core neural network architecture.
2Ease of manufacture
If existing single-person activity recognition approaches are applied to coactivities, then the system maintains simplicity in implementation, but loses the ability to utilize interaction information necessary for identifying coactivities
Solution Approach 1:
The system uses a universal key point extraction and encoding approach that works for both single-person and multi-person scenarios. The same key point detection algorithm and distance calculation method are applied regardless of the number of people detected. This multi-functional approach maintains implementation simplicity while automatically capturing interaction information when multiple people are present, as the encoded representation naturally includes inter-person distances.
3Measurement precision
If distance measurements between all key points of multiple human bodies are calculated to generate encoded representation, then coactivity classification accuracy is improved, but computational complexity increases
Solution Approach 1:
The system performs preliminary action by pre-defining a fixed set of key points on the human body (e.g., 18 specific locations). This pre-specification allows for efficient computation, as the system only needs to calculate distances between these predetermined key points rather than considering all possible point pairs. The encoded representation is generated in advance through systematic distance calculations, reducing computational complexity during actual classification.
Data Source
AI summary
Methods, systems and processor-readable media for classifying human coactivity performed jointly by two humans shown in an image or a sequence of frames of a video. A 2D convolutional neural network is used to identify key points on the human body, such as human body joints, visible within the image or within each frame, for each of the two people performing the coactivity. An encoded representation of the key points is created for each image or frame, the encoded representation being based on distances between the key points of the first person and key points of the second person. The encoded representation for the image, or a concatenated volume of the encoded representations of the frames, is processed by a fully-connected neural network trained to classify the coactivity.