Dual Sensor Video Capture for High Frame Rate Low Light
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current media capture devices face challenges in achieving a high frame rate for video content due to limitations in exposure time, leading to jerky and discontinuous video when illumination is inadequate, as longer exposure times are required for clearer details, which reduces the frame rate.
Innovation Solution
The method involves using a combination of color and panchromatic image sensors to capture frames, where the panchromatic sensor captures a greater number of frames due to higher sensitivity, allowing for the generation of color frames at a higher frame rate by processing and aligning frames from both sensors, potentially doubling or tripling the frame rate compared to single color sensor capture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a longer exposure time is used to capture clearer video details in low illumination, then the image quality and signal-to-noise ratio are improved, but the frame rate decreases resulting in jerky and discontinuous video
Solution Approach 1:
The patent divides the image sensor into two distinct types: color image sensors that capture color information at a lower frame rate with longer exposure times, and panchromatic image sensors that capture luminance information at a higher frame rate with shorter exposure times. This segmentation allows each sensor type to be optimized for its specific function, resolving the contradiction between image quality and frame rate by capturing both high-quality color data and high-frame-rate motion data separately and combining them during processing.
2Productivity
If a higher frame rate is achieved by reducing exposure time, then the video continuity is improved, but the image quality and signal-to-noise ratio deteriorate
Solution Approach 1:
The patent segments the capture function between two sensor types: color sensors optimized for quality with longer exposures, and panchromatic sensors optimized for speed with shorter exposures. The panchromatic sensor's higher frame rate capability provides the temporal continuity, while the color sensor provides the quality, and both are integrated in processing.
Solution Approach 2:
The panchromatic image sensor acts as an intermediary that captures high-frame-rate luminance information that bridges the temporal gaps between color sensor frames. This intermediary sensor enables the system to achieve high frame rates without sacrificing the color information quality captured by the dedicated color sensors.
3Loss of information
If color image sensors are used to capture video content, then color information is obtained, but the frame rate is limited due to longer exposure time requirements
Solution Approach 1:
The patent segments the information capture between color sensors (capturing chrominance at lower frame rates) and panchromatic sensors (capturing luminance at higher frame rates). This segmentation allows the system to maintain complete color information from the color sensors while achieving high frame rates through the panchromatic sensor's faster capture capability.
Solution Approach 2:
The patent merges the outputs of two different sensor types during image processing: combining the color information from color image sensors with the high-frame-rate luminance information from panchromatic image sensors. This merging produces final video frames that have both accurate color representation and high frame rate, resolving the contradiction between color information completeness and frame rate.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances video quality by increasing the frame rate, reducing jerky motion, and improving signal-to-noise ratio, especially in low-light conditions, while maintaining or improving image quality without compromising exposure times.
Implementation Method 1
panchromatic sensor pixels associated with the panchromatic image sensor have sensitivity to incident light which is greater than or equal to twice the sensitivity of the color sensor pixels
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
In accordance with an example embodiment a method, apparatus and computer program product are provided. The method comprises receiving a first set of frames corresponding to a scene captured by a color image sensor and a second set of frames corresponding to the scene captured by a panchromatic image sensor. A number of frames in the second set is greater than a number of frames in the first set. The method further comprises generating color frames corresponding to the scene at a frame rate higher than a frame rate of the first set of frames based at least in part on processing of the first set of frames and the second set of frames.