Event-Based Vision and Spiking Networks for Microsecond Motion Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing motion estimation systems, such as CNNs and LiDAR-based systems, struggle with insufficient temporal resolution for high-velocity object detection in rapidly changing environments, limiting their effectiveness in applications like autonomous driving and drone control.
Innovation Solution
A spiking neural network (SNN) is created by transferring parameters from an artificial neural network (ANN) trained with real-world motion data from a reference sensor, using an event-based vision sensor (EVS) to achieve high temporal resolution and accuracy in motion prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional frame-based cameras and CNNs are used for velocity estimation, then object detection accuracy is sufficient, but temporal resolution is limited to hundreds of Hz due to high data rates and processing time
Solution Approach 1:
The patent replaces conventional frame-based camera systems with event-based vision sensors (EVS) that operate on a different principle - instead of capturing complete frames at fixed intervals, EVS pixels independently generate events asynchronously when brightness changes occur. This substitution eliminates the mechanical/frame-based constraint and enables microsecond-level temporal resolution while maintaining velocity estimation accuracy through the spiking neural network processing pipeline
2Productivity
If LiDAR-based systems are used for object detection, then detection rate reaches ~200fps, but prediction frequency remains in the order of hundreds of Hz due to complex Recurrent Neural Network architecture
Solution Approach 1:
The patent substitutes complex Recurrent Neural Networks with spiking neural networks that are naturally suited for processing asynchronous event streams from EVS. This replacement simplifies the architecture by eliminating the need for complex temporal modeling while achieving higher prediction frequencies that match the microsecond-level temporal resolution of the event-based sensor input
3Productivity
If standard cameras operate at high framerates to improve temporal resolution, then data rates become unmanageably high, limiting practical framerate to hundreds of Hz
Solution Approach 1:
The patent extracts only the relevant information from the visual scene by having individual pixels generate events independently when brightness changes occur, rather than capturing and transmitting complete frames. This extraction approach transmits only the essential motion information at microsecond resolution, reducing data rates from megabytes per second (frame-based) to kilobytes per second (event-based) while achieving thousands of Hz effective framerate
Solution Approach 2:
The patent segments the image sensor into independently operating pixels, each capable of detecting brightness changes and generating events autonomously. This segmentation eliminates the frame-based synchronization constraint and allows parallel event generation across the sensor array, enabling high temporal resolution with manageable data rates through the event-based communication protocol
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present disclosure relate to an event-based imaging system, a method and spiking neural network therefor, a training system for creating such a spiking neural network, a vehicle, a drone, and a robotic system. The method comprises obtaining event data of an object in the environment of the event-based imaging system using an event- based vision sensor (EVS) and obtaining motion data of the object using a reference sensor. Further, the method provides for applying the event data and the motion data for training an artificial neural network (ANN) to characterize and/or predict a motion of the object. The method also comprises creating, based on the trained ANN, a spiking neural network (SNN) configured to characterize and/or predict the motion of the object.