Multi-Object Tracking With Joint Detection and Temporal Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multiple object tracking systems face challenges in achieving high accuracy due to factors like small object size, occlusion, diverse camera angles, unpredictable movement, and noise, with existing methods often decomposing into independent modules that can negatively affect overall performance.
Innovation Solution
An end-to-end deep learning model, named JDAT, is proposed for multi-object tracking, utilizing 3D-CNN, Video Transformer, or 2D-CNN with Transformer Encoder for spatial-temporal feature extraction, followed by a multi-task model for object detection and association, and a Temporal Aggregation Module for tracking state updates, trained with a Collective Average Loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If multiple independent sub-modules are used for object detection, re-identification, and association, then each module can be optimized independently with independent loss functions, but the overall accuracy decreases due to accumulated errors and cross-interaction between modules
Solution Approach 1:
The patent merges object detection, re-identification, and association into a single unified deep learning model that processes video frames end-to-end. This integration eliminates the interfaces between independent modules, preventing error accumulation and cross-interaction issues while maintaining the functional capabilities of each sub-task through shared neural network components and joint loss optimization.
2Device complexity
If object detection module processes single static frames, then computational complexity is reduced, but temporal and motion features between consecutive frames are not exploited
Solution Approach 1:
The unified model performs preliminary extraction of both spatial and temporal features from consecutive video frames before performing object detection and association. By processing temporal information in advance through convolutional operations on frame sequences, the model prepares enriched feature representations that incorporate motion cues, making the subsequent detection more accurate without adding significant computational burden at inference time.
3Device complexity
If re-identification module uses local features from cropped image patches, then computational load is reduced, but global context such as relative position and interaction between objects is not considered
Solution Approach 1:
The unified model employs a multi-functional architecture where the same neural network components serve multiple purposes: extracting local features for re-identification, capturing global context through receptive field expansion, and modeling object interactions. This is achieved through stacked convolutional layers that progressively aggregate spatial information and attention mechanisms that capture contextual relationships, allowing a single model to perform multiple functions without requiring separate specialized modules.
4Reliability
If object association module uses handcrafted motion model based on Kalman filter, then performance is good under high frame rate and linear motion, but effectiveness decreases in complex scenarios such as low FPS, complex motion or nonlinear trajectory
Solution Approach 1:
The patent replaces the handcrafted Kalman filter motion model with a learnable motion model implemented through neural network layers. This learned model adapts its parameters automatically during training to match the specific characteristics of the video data, including frame rate, motion patterns, and trajectory complexity. The flexible parameterization allows the model to handle diverse scenarios from simple linear motion to complex nonlinear trajectories without requiring manual adjustment of motion model parameters.
Data Source
AI summary
A method for multi-object tracking from video. The method includes the following steps: (1) Capturing frames from the streaming source and preprocess the data; (2) Extract video features with three choices: a 3D-CNN backbone followed by a Transformer Encoder, a Video Transformer Encoder, a 2D-CNN Encoder with a stack of frames as input followed by a Transformer Encoder; (3) Multi-object tracking using a new end-to-end multi-task deep learning model named JDAT (Joint Detection Association Transformer), then post-processing and updating tracking state with Temporal Aggregation Module (TAM). The deep learning models in step 2 and step 3 are trained simultaneously end-to-end with a loss function that is accumulated over multiple timesteps (Collective Average Loss—CAL). Also, the model can be pretrained with weakly labeled image dataset in a self-supervised learning manner first, then finetuned on supervised video datasets with full tracking labels.