Salient Frame Detection for Real-Time Repetition Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image data analysis techniques struggle with real-time prediction of user activities and motion, particularly in identifying salient frames and accurately tracking repetitions of user actions.
Innovation Solution
A system that processes image data to predict user activities in real-time by determining action, progress, and saliency scores on a per-frame basis, using a network trained to analyze body poses and identify salient frames for activities such as exercises.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a video sequence of frames is fed into a network to predict user activity, then the prediction accuracy is improved, but the real-time prediction capability deteriorates
Solution Approach 1:
The patent segments the video processing task by identifying and focusing only on salient frames rather than processing the entire video sequence. The system divides the video into discrete frames, evaluates each frame for saliency based on motion and pose changes, and processes only those frames that contain meaningful activity information. This segmentation approach maintains prediction accuracy while significantly reducing computational load to enable real-time processing.
Solution Approach 2:
The patent extracts and processes only the essential components of video data - specifically salient frames that contain meaningful activity information. By using saliency detection to extract relevant frames and discarding redundant frames, the system achieves accurate activity prediction without the computational burden of processing entire video sequences, thereby enabling real-time prediction.
2Reliability
If the entire video sequence is processed to predict user activity, then comprehensive activity recognition is improved, but computational efficiency deteriorates
Solution Approach 1:
The system segments the video sequence into individual frames and applies saliency detection to identify only those frames containing meaningful activity information. This segmentation allows comprehensive activity recognition by ensuring all relevant moments are captured, while simultaneously improving computational efficiency by processing only a subset of frames rather than the entire sequence.
Solution Approach 2:
The patent applies partial action by processing only the necessary portion of the video data - specifically salient frames - rather than the entire sequence. The saliency detection mechanism identifies frames that exceed a threshold of importance, allowing the system to achieve comprehensive activity recognition with partial processing, thereby significantly improving computational efficiency.
3Measurement precision
If repetition counting is performed on all frames, then accuracy in tracking user actions is improved, but processing time increases
Solution Approach 1:
The patent segments the repetition counting task by applying it only to salient frames identified through saliency detection, rather than processing all frames. This segmentation maintains tracking accuracy by focusing on frames that contain meaningful action information, while dramatically reducing processing time by excluding redundant frames from the counting process.
Solution Approach 2:
The system extracts salient frames using motion and pose-based saliency detection, then performs repetition counting only on these extracted frames. This approach ensures accurate tracking of user actions by processing all relevant moments, while minimizing processing time by excluding frames that do not contribute to action recognition.
Data Source
AI summary
Determining characteristics of user motion is described. The technique includes capturing a series of frames of a user performing a motion and determining progress prediction and saliency scores for each of a set of candidate actions based on the features of the frames. The progress prediction score and saliency score are determined based on features of the current frame and one or more prior frames. The progress prediction value is determined and used to track repetitions of the user motion. Upon detecting the repetition has completed, salient frames are identified based on the saliency scores.