Event Camera Spiking Neural Network Target Tracking in High-Dynamic Scenes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing target tracking methods based on correlation filtering and deep learning face limitations in feature extraction, real-time performance, and sensitivity to lighting and dynamic scenes, while event-based cameras offer asynchronous event streams with high dynamic range and low latency.
Innovation Solution
A spiking neural network-based target tracking method using an event camera, involving data stream acquisition, synchronous event frame generation, and a twin network trained with a gradient substitution algorithm for accurate target positioning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning-based target tracking is used, then feature extraction ability and tracking accuracy are improved, but computational complexity increases and real-time performance deteriorates
Solution Approach 1:
The patent segments the tracking task into two distinct components: a deep learning-based offline training phase that extracts robust feature representations, and a lightweight online correlation filtering phase that performs real-time tracking. This segmentation allows the complex feature extraction to be performed once during training, while the runtime system uses simpler operations to maintain real-time performance.
Solution Approach 2:
The patent performs preliminary action by pre-training deep learning models offline to learn optimal feature representations and target characteristics. These pre-learned features are then stored and reused during real-time tracking, eliminating the need to perform complex deep learning computations during the time-critical tracking phase.
2Speed
If frame-rate camera is used for target tracking, then continuous image data is obtained, but latency increases and dynamic range is limited
Solution Approach 1:
The patent employs periodic action by accumulating asynchronous events over fixed time intervals (e.g., 10ms) to synthesize equivalent frame-rate images. This approach maintains the low-latency advantages of event cameras while providing periodic updates at standard frame rates, enabling compatibility with traditional tracking algorithms.
Solution Approach 2:
The patent introduces an intermediary conversion process that transforms asynchronous event streams into synchronous frame-equivalent images. This intermediary representation allows the system to leverage both the temporal precision of event cameras and the algorithmic compatibility of frame-based processing methods.
3Device complexity
If conventional tracking methods are used in high dynamic scenes, then simplicity is maintained, but tracking accuracy deteriorates under severe lighting conditions
Solution Approach 1:
The patent changes the fundamental parameter of how visual data is represented and processed. Instead of using intensity-based frame images that struggle with high dynamic range, the system uses asymptotic event streams that naturally encode brightness changes and temporal information, enabling accurate tracking in severe lighting conditions while maintaining relative simplicity.
Data Source
AI summary
A target tracking method and a target tracking system of a spiking neural network based on an event camera are provided. The method includes: acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data; dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution; training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image and a complete image as a searched image; and tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image. The twin network includes a feature extractor and a cross-correlation calculator.