Multi-Parity FSR Mechanism for High Frame Rate Video Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Flexible Sub-sampled Readout (FSR) techniques for imaging devices suffer from low resolution limitations, edge artifacts, flicker effects, and motion artifacts, and are unable to process sub-sampled Bayer patterns effectively, limiting high frame rate and high resolution video capture.
Innovation Solution
Implementing a multi-parity FSR mechanism that spatially and temporally samples full frame Bayer data using a zig-zag sampling pattern, allowing for high frame rate and high resolution video reconstruction, with the option to apply this in either pre-Image Signal Processor (ISP) or post-ISP FSR reconstruction based on bandwidth capacity, and enabling selective high frame rate capture for Regions of Interest (ROI) while maintaining normal frame rates for the rest of the video.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If pixel readout modes such as sub-sampling or binning are used to achieve higher frame rates, then frame rate is improved, but spatial resolution deteriorates
Solution Approach 1:
The patent divides the full Bayer frame into multiple parity fields (e.g., 4 parity fields) through spatial sub-sampling. Each parity field contains a subset of pixels from the original frame, allowing the system to capture multiple parity fields sequentially at high frame rates and then reconstruct the full-resolution video by combining these segmented parity fields temporally and spatially.
Solution Approach 2:
The patent transitions from spatial sub-sampling alone to a combination of spatial and temporal sub-sampling. By introducing the time dimension, the system captures multiple low-resolution parity fields at high frame rates and reconstructs them into high-resolution video frames, effectively trading spatial information for temporal information and then recovering spatial resolution through reconstruction algorithms.
2Speed
If Flexible Sub-sampled Readout (FSR) techniques are used to achieve very high frame rates, then frame rate is improved, but resolution deteriorates and edge artifacts are introduced
Solution Approach 1:
The patent introduces an intermediary reconstruction process that acts as a mediator between the sub-sampled parity fields and the final video output. This reconstruction pipeline includes multiple stages (de-mosaicing, upsampling, temporal interpolation, and artifact reduction) that progressively restore resolution and eliminate artifacts, transforming the low-resolution parity fields into high-resolution video frames.
Solution Approach 2:
The patent performs preliminary capture of multiple parity fields at high frame rates before reconstruction. By capturing all necessary parity field data in advance and storing it in buffer memory, the system prepares the raw sub-sampled data for subsequent reconstruction, ensuring that all information needed for high-resolution recovery is available before the reconstruction process begins.
3Speed
If existing FSR techniques are used for sub-sampling in Bayer domain, then frame rate is improved, but edge artifacts and flicker effects are introduced
Solution Approach 1:
The patent converts the potentially harmful sub-sampling operation into a beneficial process by carefully designing the parity field sampling pattern and reconstruction algorithm. Instead of simply discarding pixels, the system strategically samples specific pixel patterns across multiple parity fields and uses this structured sub-sampling information to reconstruct the full frame with reduced artifacts, turning the limitation into an opportunity for optimized capture.
Solution Approach 2:
The patent implements dynamic reconstruction that adapts to scene content and motion characteristics. The reconstruction pipeline dynamically adjusts parameters such as temporal interpolation strength, upsampling filters, and artifact reduction intensity based on the captured parity fields and detected motion, allowing the system to optimize quality and minimize artifacts for each specific scene condition.
4Device complexity
If traditional image sensor pipeline is used to process sub-sampled Bayer pattern, then processing simplicity is maintained, but processing capability deteriorates
Solution Approach 1:
The patent segments the processing pipeline into distinct functional stages: parity field capture, de-mosaicing, upsampling, temporal interpolation, and artifact reduction. Each stage handles a specific aspect of the reconstruction process, allowing the system to process sub-sampled Bayer patterns effectively while maintaining organized and manageable complexity through modular processing blocks.
Data Source
AI summary
Methods and systems for reconstructing a high frame rate high resolution video in a Bayer domain, when an imaging device is set in a Flexible Sub-Sampled Readout (FSR) mode are described. A method provides the FSR mode, which utilizes a multiparty FSR mechanism to spatially and temporally sample the full frame Bayer data. The multi parity FSR utilizes a zigzag sampling that assists reconstruction of motion compensated artifact free high frame rate high resolution video with full frame size. The method includes reconstructing the high frame rate high resolution video using the plurality of parity fields generated. The reconstruction is based on a FSR reconstruction mechanism that can be a pre-Image Signal Processor (ISP) FSR reconstruction or a post-ISP FSP reconstruction based on bandwidth capacity of an ISP used by the imaging device.