Few-Shot Video Action Recognition With DML Trajectories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition technologies in computer vision require large amounts of labeled training data, making it costly and difficult to learn new action classes, and there is a lack of effective methods for few-shot learning in video classification and temporal action detection.
Innovation Solution
A method using Distance Metric Learning (DML) to encode long-term sub-action connectivity into an embedding space, represented as trajectories, with a novel loss function incorporating diversity and sub-action discrimination, trained on a few fully connected layers to achieve state-of-the-art results in few-shot benchmarks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large amounts of labeled training data are used for action recognition, then classification accuracy is improved, but data collection cost and processing time increase
Solution Approach 1:
The patent segments actions into sub-actions and represents them as trajectories in embedding space. This segmentation allows the system to learn from fewer examples by breaking down complex actions into manageable components that can be recognized with limited data
Solution Approach 2:
The patent pre-trains the embedding network on large datasets to learn general action representations, then fine-tunes on少量 labeled data. This preliminary action of pre-training allows the system to achieve high accuracy without requiring extensive labeled data for each specific task
2Measurement precision
If traditional deep learning methods are used for action recognition, then performance on large datasets is improved, but adaptability to new action classes with limited data deteriorates
Solution Approach 1:
The patent introduces an embedding network as an intermediary that maps actions to a continuous embedding space. This intermediary representation allows the system to generalize to new action classes by comparing trajectories in the embedding space, enabling few-shot learning while maintaining high recognition performance
Solution Approach 2:
The patent changes the parameter representation from discrete action labels to continuous embedding vectors. This parameter change allows for smooth generalization to new classes and enables the system to adapt to few-shot scenarios by learning the underlying manifold structure of action space
3Measurement precision
If fine-grained action discrimination is achieved, then classification precision is improved, but computational complexity increases
Solution Approach 1:
The patent moves action representation from the original feature space to an embedding space with learned metrics. This dimensionality change allows fine-grained discrimination through trajectory comparison in the embedding space, achieving precise classification while managing computational complexity through efficient distance calculations
Data Source
AI summary
Approaches presented herein enable action recognition. More specifically, a plurality of video segments having one or more action representations is received. One or more sub-action representations in the plurality of video segments are learned. An embedding in a space of a distance metric learning (DML) network for each of the one or more sub-action representations is determined. A set of respective trajectory distances between each of the one or more sub-action representations and one or more class representatives in the space of the DML network based on the embedding is computed, and the one or more action representations based on the set of respective trajectory distances are classified.