Event-Based Camera Action Recognition via Surrogate RGB Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition methods are computationally expensive and reliant on noisy optical flow measurements, which are sensitive to lighting conditions, making them inefficient and robustness-challenged for real-time applications.
Innovation Solution
An efficient method using an event-based camera to recognize human actions by constructing surrogate RGB images from asynchronously transmitted pixel locations, computing motion event features, and employing a two-stream deep neural network architecture with fine-tuned CNNs and linear SVM for action classification, tolerant to varying lighting conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If optical flow is used for action recognition, then measurement precision is improved, but computational time increases and robustness to lighting conditions deteriorates
Solution Approach 1:
The patent extracts only the essential motion information from the visual data by using event-based cameras that detect and transmit only pixels with intensity changes. This eliminates the need for computing full optical flow fields, removing the problematic component while retaining the useful motion signal for action recognition.
Solution Approach 2:
The patent replaces the traditional optical flow computation mechanism with an event-based detection mechanism. Instead of calculating dense optical flow fields through computationally intensive algorithms, the system uses asynchronous event streams that directly encode motion information, substituting a mechanical computation process with a more efficient sensing approach.
2Measurement precision
If traditional action recognition methods are used, then action recognition accuracy is improved, but computational time and computational resources increase
Solution Approach 1:
The patent extracts only the essential motion information from the visual data by using event-based cameras that detect and transmit only pixels with intensity changes. This eliminates the need for computing full optical flow fields, removing the problematic component while retaining the useful motion signal for action recognition.
Solution Approach 2:
The patent changes the fundamental parameters of motion representation from dense optical flow vectors to sparse event coordinates (pixel location, time, polarity). This parameter transformation reduces the data dimensionality and computational complexity while preserving the essential motion patterns needed for accurate action recognition.
3Loss of information
If optical flow is used for motion measurement, then motion information is captured, but sensitivity to lighting conditions increases
Solution Approach 1:
The patent converts the harmful sensitivity to lighting changes into a beneficial feature by using event-based cameras that detect intensity changes as the core measurement principle. What was previously a source of noise and error in optical flow (lighting variations) becomes the fundamental signal for detecting motion, as events are triggered by any intensity change regardless of absolute lighting conditions.
Data Source
AI summary
A method for recognizing an action captured on an event-based camera includes the steps of receiving asynchronously transmitted pixel locations which have changed intensity via an address-event bus; stacking a plurality of frames of received pixel location to form a surrogate RGB image where the channels represent individual frames; inputting the surrogate RGB image into a first convolutional neural network; and extracting feature f1 from a last fully connected layer of the convolutional neural network to obtain an action classification of the action, thereby recognizing the action.