Semi-supervised Video Action Recognition via Soft Label Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently training machine learning models for frame-level action segmentation and labeling in long-duration data, such as videos, due to the high cost and time required for annotating vast amounts of data, especially in domains like manufacturing where data can exceed ten-thousand hours.
Innovation Solution
A semi-supervised training approach is employed, where only a small fraction of annotated videos are used, leveraging domain knowledge to learn from a sizeable corpus of unlabeled data, utilizing multi-stream temporal convolution networks and sequence-to-sequence models to generate soft labels and reduce annotation costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full supervision with extensive annotated data is used for training, then model accuracy is improved, but annotation cost and time increase significantly
Solution Approach 1:
The system uses the model's own predictions on unlabeled data to generate soft labels, which are then used to supervise the model. This self-generated supervision eliminates the need for extensive manual annotation while still providing guidance for accurate learning, resolving the contradiction between model accuracy and annotation time
Solution Approach 2:
Soft labels serve as an intermediary between the model's raw predictions and the final annotated data. These soft labels provide graded supervision signals that guide the model learning process without requiring full manual annotation, thereby maintaining accuracy while reducing annotation effort
2Loss of information
If full annotation of all video data is performed, then labeling completeness is improved, but cost and resource requirements increase
Solution Approach 1:
The system performs partial annotation by generating soft labels for only a subset of unlabeled data rather than annotating everything. This partial supervision approach maintains sufficient labeling completeness for effective training while dramatically reducing the human resources and costs required
Solution Approach 2:
The model generates its own supervision signals through soft labels, eliminating the need for extensive external annotation resources. This self-service mechanism ensures labeling completeness is achieved through automated processes rather than manual effort, reducing resource requirements
3Reliability
If more annotated data is collected and used for training, then training robustness is improved, but scalability to new domains decreases
Solution Approach 1:
The semi-supervised framework with soft label generation is designed to be domain-agnostic and can be applied universally across different domains. The model learns domain-specific patterns from unlabeled data in each new domain without requiring re-annotation, maintaining training robustness while enabling easy scalability to new domains
Solution Approach 2:
The model adapts to new domains by generating its own soft labels from unlabeled domain-specific data, eliminating the need for domain-specific annotated datasets. This self-adaptation mechanism ensures training robustness is maintained across domains while dramatically improving scalability
Data Source
AI summary
Systems, apparatuses, and methods include technology that generates final frame predictions for a first plurality of frames of a video, where the first plurality of frames is associated with unlabeled data. The technology predicts an ordered list of actions for the first plurality of frames based on the final frame predictions, and temporally aligning the ordered list of actions to the final frame predictions to generate labels.