Video Processing Using Chronological Stage Sequences for Online Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for temporal action recognition and segmentation in human or machine-assistive systems lack foresight, making them ineffective for online action recognition due to the absence of future context, especially when variations in actions, order, types, and duration occur.
Innovation Solution
A video processing system that utilizes a chronological stage sequence of tasks, generated by a first trained model, to inform a second trained model for recognizing actions, segmenting videos, and predicting future actions, incorporating self-supervised learning and attention layers to handle variations and provide recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current temporal action recognition solutions are used, then action recognition can be performed, but the system lacks foresight and cannot effectively handle online action recognition due to absence of future context
Solution Approach 1:
The patent applies preliminary action by pre-defining chronological stage sequences that represent possible future task developments. The system prepares multiple potential stage sequences in advance, allowing it to anticipate future actions and provide foresight in online action recognition, thereby resolving the information loss about future context.
2Reliability
If chronological stage sequence is introduced to provide foresight, then online action recognition accuracy improves, but system complexity increases due to multiple trained models and processing layers
Solution Approach 1:
The patent segments the action recognition system into distinct functional components: a first trained model for generating chronological stage sequences, a second trained model for processing videos, and attention layers for integrating temporal information. This segmentation allows each component to specialize in specific tasks, improving overall accuracy while making the complex system more manageable and interpretable.
Solution Approach 2:
The patent introduces chronological stage sequences as an intermediary representation between raw video input and action recognition output. This intermediary structure serves as a bridge that organizes temporal information in a meaningful sequence, allowing the system to leverage future context without directly processing all possible future scenarios, thus managing complexity.
3Adaptability or versatility
If variations in actions, order, types, and duration are accommodated, then adaptability improves, but measurement precision becomes more difficult due to increased variability
Solution Approach 1:
The patent applies dynamics by making the chronological stage sequences adaptable rather than fixed. The system can dynamically adjust to variations in action types, orders, and durations while maintaining the structural framework of stage sequences. This dynamic approach allows the system to handle diverse scenarios while preserving measurement precision through the organized temporal structure.
Solution Approach 2:
The patent utilizes parameter changes by allowing the chronological stage sequences to be modified based on observed variations in actions. The system can adjust temporal parameters, stage definitions, and sequence structures to accommodate different action types and orders, thereby maintaining adaptability while preserving measurement precision through structured parameter management.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a technical solution to process a video that captures actions to be performed for completing a task based on a chronological sequence of stages within the task. An example system may identify an action sequence from an instruction for the task. The system inputs the action sequence into a trained model (e.g., a recurrent neural network), which outputs the chronological sequence of stages. The RNN may be trained through self-supervised learning. The system may input the video and the chronological sequence of stages into another trained model, e.g., a temporal convolutional network. The other trained model may include hidden layers arranged before an attention layer. The hidden layers may extract features from the video and feed the features into the attention layer. The attention layer may determine attention weights of the features based on the chronological sequence of stages.