Hybrid Frame Event Sensor Image Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional image and video acquisition systems face challenges such as motion artifacts, low light sensitivity, and high dynamic range issues, which are not effectively addressed by existing solutions that often require complex computational processes.
Innovation Solution
The method combines frame-based and event-based sensors to obtain a reference image frame and a stream of events synchronized during the exposure duration, allowing for the derivation of a corrected image frame that removes artifacts such as wobble, skew, and temporal aliasing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If rolling shutter readout is used to make image sensors smaller and more sensitive, then sensor size and cost are reduced, but motion artifacts occur during frame capture
Solution Approach 1:
An event-based sensor acts as an intermediary between the rolling shutter readout and the final image output. The event-based sensor detects brightness changes asynchronously and provides motion information that is used to correct artifacts in the frame-based sensor output, thereby mediating the harmful effects of rolling shutter readout while preserving its size and cost advantages
2Illumination intensity
If exposure duration is increased to capture more light, then low light sensitivity is improved, but motion blur increases
Solution Approach 1:
The event-based sensor provides real-time feedback about brightness changes and motion in the scene. This feedback is used to dynamically adjust the exposure duration of the frame-based sensor, allowing longer exposures in static low-light conditions while automatically shortening exposure when motion is detected, thereby maintaining image sharpness while improving low light sensitivity
3Illumination intensity
If ISO gain is increased to compensate for smaller input signal, then low light sensitivity is improved, but sensor noise is amplified
Solution Approach 1:
The event-based sensor serves as an intermediary that detects motion and brightness changes without being subject to the same noise amplification issues. Its clean event data is used to correct the frame-based sensor output, providing a noise-free path for motion information that bypasses the need for high ISO gain
4Illumination intensity
If multiple images are combined to produce HDR image, then dynamic range is improved, but computational complexity increases
Solution Approach 1:
The event-based sensor provides a continuous stream of brightness change information that naturally captures the full dynamic range of the scene over time. This continuous data flow allows HDR-like results to be achieved through a single integrated processing pipeline rather than requiring multiple discrete image captures and complex alignment operations, thereby reducing computational complexity while maintaining dynamic range improvement
Data Source
AI summary
An image enhancement method, comprising:obtaining a reference image frame from of a scene, by a frame-based sensor, wherein the reference image contains artifacts and has an exposure duration;obtaining a stream of events, by an event-based sensor which is synchronized with the frame-based sensor, at least during the exposure duration, wherein the events encode brightness changes of the scene corresponding to the reference image; andderiving a corrected image frame without the artifacts from the reference image frame using the stream of events.