Video Object Detection Using Patch Clustering for Real-Time Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile terminals with limited processor performance face challenges in executing deep learning models for high-definition video analysis due to increased delay and tracking errors, especially with high-quality input frames.
Innovation Solution
A method involving a patch recommendation unit that identifies tracking-failure and new-object patches in video frames, followed by a patch aggregation unit that generates clusters for improved object detection, utilizing machine learning to predict tracking failures and adjust patch sizes, and a patch object detection unit for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep learning model is executed for each frame of video to detect objects accurately, then object detection precision is improved, but processing time increases and real-time performance deteriorates
Solution Approach 1:
The video processing is segmented into key frames and non-key frames. Deep learning models are executed only on key frames, while non-key frames use tracking-based detection. This segmentation reduces the frequency of computationally intensive deep learning executions while maintaining detection accuracy through tracking continuity across frames.
Solution Approach 2:
The system performs periodic deep learning inference at key frame intervals rather than continuously on every frame. This periodic execution pattern reduces overall processing time and computational load while maintaining detection accuracy through the periodic updates provided by key frame analysis combined with continuous tracking.
2Measurement precision
If a deep learning model is executed on high-quality input frames to improve detection accuracy, then object detection precision is improved, but processing delay increases
Solution Approach 1:
Frames are segmented into key frames and non-key frames based on quality metrics. High-quality frames are identified as key frames and trigger deep learning execution, while lower-quality frames use lighter tracking methods. This selective approach ensures deep learning is applied only when it provides maximum benefit, reducing unnecessary processing delays.
Solution Approach 2:
The system dynamically changes processing parameters based on frame quality. When frame quality is high, deep learning models are activated for accurate detection. When quality is lower, the system switches to tracking-based detection with reduced computational requirements. This parameter adaptation maintains detection precision while minimizing processing delay across varying frame qualities.
3Reliability
If tracking model is executed for every frame to maintain continuous tracking, then tracking reliability is improved, but accumulation of tracking errors increases over time
Solution Approach 1:
The system performs preliminary deep learning detection on key frames to establish accurate initial object positions and characteristics before tracking continues through subsequent frames. This preliminary accurate detection resets and corrects potential tracking errors before they can accumulate, maintaining both tracking reliability and accuracy over extended video sequences.
Solution Approach 2:
The system uses feedback from periodic deep learning detections on key frames to correct and reset tracking states. When a key frame is processed through deep learning, the detected object positions and characteristics feed back into the tracking system, correcting any drift or errors that have accumulated during the tracking interval and maintaining long-term tracking accuracy.
4Productivity
If processor performance is increased to handle high-definition video analysis in real time, then processing speed is improved, but device complexity and cost increase
Solution Approach 1:
The computational workload is segmented between two types of processing: lightweight tracking operations executed on every frame, and intensive deep learning inference executed only on key frames. This segmentation allows mobile processors with limited performance to handle real-time video analysis by distributing computational demands across different processing capabilities and time intervals.
Solution Approach 2:
Instead of applying full deep learning processing to every frame, the system applies partial processing (tracking) to all frames and reserves full deep learning action for only the necessary key frames. This partial action approach achieves real-time performance on mobile devices by performing sufficient detection and tracking without the excessive computational burden of frame-by-frame deep learning.
Data Source
AI summary
Provided is a video analysis terminal including a patch recommendation unit configured to recommend tracking-failure patches and new-object patches in a current frame of a video image, and a patch aggregation unit configured to generate a first patch cluster by collecting the tracking-failure patches recommended in the current frame, and generate a second patch cluster by collecting the new-object patches recommended in the current frame.