Mask-Based Temporal Action Segmentation for Continual Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional temporal action segmentation models face issues such as catastrophic forgetting and background semantic shift due to new parameter additions and frame-wise classification methods, leading to performance degradation and over-segmentation errors during continual learning.
Innovation Solution
A mask-based framework device that includes a first framework model trained in a previous task and a second framework model trained in a current task, using binary action masks and class-agnostic features to preserve knowledge through pseudo-labeling and knowledge distillation, reducing errors and maintaining previous task knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional temporal action segmentation models use multi-stage architecture with iterative refinement, then classification accuracy is improved, but device complexity increases and catastrophic forgetting occurs due to new parameter additions
Solution Approach 1:
The model segments the action recognition task into two independent components: binary action mask generation (temporal segmentation) and action class classification. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining accuracy.
Solution Approach 2:
The binary action mask generator serves multiple functions: it performs temporal segmentation, provides temporal context for classification, and enables continual learning through pseudo-label generation. This multi-functionality reduces the need for separate specialized components.
2Productivity
If frame-wise classification method is used for action segment prediction, then classification speed is improved, but over-segmentation errors increase leading to performance degradation
Solution Approach 1:
The patent merges temporal segmentation and action classification into a unified framework where the binary action mask generator and class classifier work together. This combination allows temporal context to inform classification decisions, reducing over-segmentation errors while maintaining fast frame-wise processing.
Solution Approach 2:
The binary action mask serves as an intermediary between raw frame-wise predictions and final action segmentation. It provides temporal coherence and contextual information that reduces over-segmentation errors while enabling efficient frame-wise processing.
3Adaptability or versatility
If new classes are learned in continual learning setting, then model adaptability is improved, but previously learned knowledge is lost due to background semantic shift
Solution Approach 1:
The model performs preliminary action segmentation using the binary mask generator before进行分类. This preliminary temporal context preservation helps maintain previously learned knowledge when adapting to new classes, as the temporal structure information is retained and can be used to generate pseudo-labels for continual learning.
Solution Approach 2:
The patent uses pseudo-label copying from the binary action mask generator to preserve temporal context information when learning new classes. The mask predictions are copied and used as supervisory signals to maintain previously learned temporal patterns during continual learning of new action classes.
Data Source
AI summary
A mask-based framework device for continual learning of temporal action segmentation includes an interface unit configured to perform data input/output and a framework model unit configured to perform temporal action segmentation, in which the framework model unit includes a first framework model trained through a previous task and a second framework model trained through a current task from the first framework model, and each of the first framework model and the second framework model receive image data and output binary action mask information and action class classification information.