DVS Frame Auto-Labeling Using Stereo 3D Re-Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing auto-labeling methods for DVS frames are inefficient and inaccurate, failing to leverage the advantages of DVS such as low-latency, high dynamic range, and low power consumption, and often result in unnatural or blurred frames.
Innovation Solution
A method and system that combines a pair of stereo cameras with a DVS to calculate disparity frames, obtain 3D information, and use deep learning to determine object areas, re-projecting 3D points onto DVS frames for accurate auto-labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional camera video is used to record and label objects, then labeling can be performed, but the precision is lost due to inability to match DVS frame exactly to displaying monitor
Solution Approach 1:
The patent introduces a disparity frame as an intermediary element that bridges the camera frame and DVS frame. The disparity frame, calculated from stereo camera frames, provides 3D information that enables precise matching between camera and DVS frames without direct screen recording, thus maintaining both precision and efficiency
Solution Approach 2:
The patent transitions from 2D screen recording to 3D spatial mapping by calculating disparity frames and obtaining 3D information. This dimensional change allows precise frame matching through spatial coordinates rather than screen pixel matching, resolving the precision-loss problem
2Extent of automation
If deep learning model is used to directly generate labeled DVS frames from camera frames, then auto-labeling is achieved, but the generated frames are unnatural because reflection rates of different materials are treated the same
Solution Approach 1:
The patent applies local quality by using disparity information and 3D point re-projection to preserve material-specific reflection characteristics. Each pixel's labeling is determined by its specific spatial position and depth information, allowing different materials to maintain their unique optical properties rather than being treated uniformly
Solution Approach 2:
Instead of generating synthetic DVS frames, the patent copies real DVS frame data and labels it using 3D re-projection technology. This copying approach preserves the natural characteristics of actual DVS data while enabling automated labeling through spatial mapping
3Productivity
If camera frames are used to generate DVS frames, then labeling can be performed, but the advantages of DVS are wasted due to limited frame rate, motion blur, and low dynamic range
Solution Approach 1:
The patent extracts the labeling information from camera frames through 3D spatial mapping, then applies this extracted information to real DVS frames. This separation allows the labeling capability to be transferred without transferring the limitations of camera frames, preserving DVS advantages while achieving automated labeling
Solution Approach 2:
The patent performs preliminary 3D reconstruction and object detection on camera frames, then uses these pre-computed results to label DVS frames. This preliminary action enables the DVS frames to retain their high frame rate and low motion blur characteristics while still achieving accurate labeling
Data Source
AI summary
The disclosure provides a method, a system and an apparatus for auto-labeling dynamic vision sensor (DVS) frame. The method may comprise generating a pair of camera frames by a pair of cameras within an interval, and generating at least one DVS frame by a DVS within the interval. The method may further calculate a disparity frame based on the pair of camera frames, and obtain 3D information of the pair of camera frames based on the calculated disparity frame. The method may use a deep leaning model to determine an object area for auto-labeling, and may obtain 3D points based on the 3D information and the determined object area. And then, the method may re-project the 3D points towards the at least one DVS frame to generate re-projected points on the at least one DVS frame. The method may further generate at least one auto-labeled result on the at least one DVS frame by combining the re-projected points on the at least one DVS frame.