Neural Network Object Tracking with Multi-Threaded Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object detection and tracking systems using neural networks face challenges in real-time performance due to high memory bandwidth and computation costs, particularly on mobile devices, where deep learning-based detectors are computationally intensive and often slow, making real-time object tracking difficult.
Innovation Solution
A multi-threaded approach is employed, where a trained neural network runs on a low frame per second thread for detection, and lightweight object trackers run on a high frame rate thread for tracking, with a strong tracker performing lag compensation to account for the delay in neural network processing, allowing for real-time object tracking across images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a trained neural network is applied to detect objects in images, then detection accuracy is improved, but processing speed deteriorates due to high computation cost
Solution Approach 1:
The system divides object detection into two segments: a trained neural network applied periodically at low frame rates for accurate detection, and lightweight trackers applied at every frame for continuous tracking. This segmentation allows the computationally intensive neural network to operate less frequently while maintaining overall system accuracy and speed.
Solution Approach 2:
The trained neural network performs preliminary object detection at lower frame rates to identify objects of interest. These detections serve as initial conditions for subsequent lightweight trackers that continue tracking at higher frame rates, eliminating the need to re-run the full neural network at every frame.
2Measurement precision
If a trained neural network is applied to detect objects in images, then detection quality is improved, but memory bandwidth consumption increases
Solution Approach 1:
The system segments the detection pipeline so that only periodic frames undergo full neural network processing with high memory bandwidth consumption, while intermediate frames use lightweight trackers with minimal memory access. This dramatically reduces overall memory bandwidth requirements while preserving detection quality through periodic neural network updates.
3Speed
If a lightweight tracker is used to track objects at high frame rates, then tracking speed is improved, but tracking accuracy deteriorates due to lag
Solution Approach 1:
The system uses feedback from periodic neural network detections to correct and update tracker states. The neural network detections serve as ground truth references that correct cumulative tracking errors, ensuring long-term tracking accuracy while allowing the lightweight tracker to operate at high speeds between corrections.
Solution Approach 2:
The neural network performs preliminary accurate detection at periodic intervals to establish correct object positions. These preliminary detections compensate for the lag inherent in high-speed lightweight tracking, effectively resetting tracking accuracy without sacrificing frame rate.
4Loss of time
If real-time object tracking is implemented using neural networks, then tracking timeliness is improved, but device resource consumption increases
Solution Approach 1:
The tracking system is segmented into periodic neural network detection tasks and continuous lightweight tracking tasks. This segmentation enables real-time responsiveness through the lightweight trackers while minimizing resource consumption by restricting expensive neural network operations to periodic intervals rather than every frame.
Data Source
AI summary
Techniques and systems are provided for tracking objects in one or more images. For example, a trained network can be applied to a first image of a sequence of images to detect one or more objects in the first image. The trained network can be applied to less than all images of the sequence of images. A second image of the sequence of images and a detection result from application of the trained network to the first image are obtained. The detection result includes the detected one or more objects from the first image. A first object tracker can be applied to the second image using the detection result from application of the trained network to the first image. Applying the first object tracker can include adjusting one or more locations of one or more bounding boxes associated with the detected one or more objects in the second image to track the detected one or more objects in the second image. A second object tracker can also be applied to the second image to track at least one object of the detected one or more objects in the second image. The second object tracker is applied to more images of the sequence of images than the trained network and the first object tracker. Object tracking can be performed for the second image based on application of the first object tracker and the second object tracker.