Segmentation-Assisted Object Tracking for Low-Resource Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision systems require significant computational resources for accurate object tracking in video frames, making it challenging for low-resource devices to deliver high-quality live processing of video data.
Innovation Solution
Implementing a segmentation-based approach that uses a lightweight visual association model (VAM) to track objects across frames based on segmentation masks generated from a sparse set of reference frames processed by a more computationally intensive object detection model (ODM), reducing the need for full-frame processing by ODM and enhancing tracking accuracy with Kalman filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a computationally intensive object detection model (ODM) processes every video frame, then object detection accuracy is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The video processing is segmented into two parts: reference frames processed by the intensive ODM for high accuracy, and non-reference frames processed by the lightweight VAM for speed. This segmentation allows the system to achieve both accuracy and efficiency by applying different processing strategies to different portions of the video data.
Solution Approach 2:
The ODM is applied periodically to reference frames at predetermined intervals, while the VAM handles intermediate frames continuously. This periodic intensive processing combined with continuous lightweight processing resolves the contradiction between accuracy and processing speed.
2Measurement precision
If a computationally intensive object detection model (ODM) processes every video frame, then object detection accuracy is improved, but computational resources required increase significantly
Solution Approach 1:
The system segments computational workload by applying the resource-intensive ODM only to reference frames while using the efficient VAM for non-reference frames. This segmentation dramatically reduces overall computational resource consumption while preserving detection accuracy through periodic ODM processing.
Solution Approach 2:
The VAM creates a visual association copy of object appearances from reference frames to track objects in non-reference frames. This copying approach allows accurate tracking without repeatedly executing the full ODM, thereby reducing computational resource usage.
3Productivity
If a lightweight visual association model (VAM) tracks objects in all frames, then processing speed is improved, but tracking accuracy for occluded or transformed objects deteriorates
Solution Approach 1:
The ODM performs preliminary action by processing reference frames in advance to establish accurate object representations. These pre-processed reference frames serve as reliable anchors that improve tracking accuracy when the VAM encounters occluded or transformed objects in non-reference frames.
Solution Approach 2:
The system uses feedback by comparing VAM tracking results with periodic ODM detections of reference frames. When discrepancies are detected (indicating occlusion or transformation), the system can correct tracking accuracy using the more reliable ODM results, thereby maintaining both speed and reliability.
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques for segmentation-assisted detection and tracking of objects or features in videos, across images, and/or in other 2D and/or 3D visual content. The techniques include processing a plurality of frames of a video to obtain a plurality of representations of an object depicted in the video. A first subset of the plurality of representations is obtained by processing, using an object detection model, a first subset of the plurality of frames. A second subset of the plurality of representations is obtained using visual similarity of an appearance of the object in a second subset of the plurality of frames to the appearance of the object in at least one other frame of the plurality of frames. The techniques further include obtaining, using the plurality of representations, segmentation masks for the plurality of frames and performing one or more operations based on the segmentation masks.