Event-Assisted Camera Fusion for Low-Latency Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Camera systems face challenges in balancing operating speed and resolution due to increased latency and data volume, particularly in on-vehicle applications, leading to reduced detection accuracy.
Innovation Solution
A camera system incorporating an image sensor and an event-based sensor that captures visual and event data at different frequencies, with a processing unit to synchronize and fuse these frames using temporal-spatial masks to enhance detection accuracy and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image resolution and frame rate are increased, then detection accuracy is improved, but latency increases and operating speed decreases
Solution Approach 1:
The patent segments the image sensor into multiple regions with different sampling rates. High-motion regions are sampled at higher rates while low-motion regions use lower sampling rates, allowing the system to maintain detection accuracy for moving objects while reducing overall data volume and latency.
Solution Approach 2:
The patent implements dynamic frame rate adjustment where the sampling frequency of different sensor regions is dynamically changed based on detected motion. When motion is detected in a region, its sampling rate increases; when no motion is present, the rate decreases, thus balancing detection accuracy with latency reduction.
2Measurement precision
If image resolution is increased, then detection accuracy is improved, but data volume increases and processing complexity increases
Solution Approach 1:
The patent divides the image sensor into multiple regions that can be independently sampled at different rates. This segmentation allows the system to capture high-resolution data only in regions where it is needed (where motion is detected) while using lower resolution in static regions, thereby reducing overall data volume while maintaining detection accuracy where required.
Solution Approach 2:
The patent applies different sampling qualities to different regions of the sensor based on local motion characteristics. Regions with detected motion receive higher sampling rates and resolution, while regions without motion use lower sampling rates, optimizing the balance between detection accuracy and data volume reduction.
3Speed
If frame rate is increased, then operating speed is improved, but power consumption increases
Solution Approach 1:
The patent dynamically adjusts the frame rate of different sensor regions based on motion detection. When motion is detected in a region, the frame rate increases to capture the event; when no motion is present, the frame rate decreases to conserve power. This dynamic adjustment allows the system to maintain high operating speed when needed while reducing power consumption during static periods.
Solution Approach 2:
The patent implements periodic motion detection followed by conditional high-speed sampling. Instead of continuous high-frame-rate sampling, the system periodically checks for motion and only increases sampling rate when motion is detected, creating a periodic action pattern that reduces average power consumption while maintaining the capability for high-speed operation when required.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system achieves faster response, increased dynamic range, reduced memory and power consumption, and improved operating speed and image resolution, enhancing detection accuracy in on-vehicle systems.
Implementation Method 1
event sensors or event sensing pixels that record variations in light intensity in a targeted scene
Data Source
AI summary
The present disclosure provides a camera system and an event-assisted image processing method. The camera system includes an image sensor, an event-based sensor, and a processing unit. The image sensor is configured to capture visual images of a targeted scene to obtain image sensing frames with a first frequency. The event-based sensor is configured to capture event data of the targeted scene to obtain event frames with a second frequency higher than the first frequency. The processing unit is configured to: receiving the image sensing frames within a predetermined time period; accumulating the event frames within the predetermined time period; generating a temporal-spatial mask indicating interested areas for the event frames; determining geometric features in the temporal-spatial masks; synchronizing the image sensing frames and the event frames at timestamps within the predetermined time period; and fusing the temporal-spatial mask with the image sensing frames to obtain a masked visual image.