Super-Resolution Image Generation Using Sparse Bayer Sensor and IMU Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed-reality systems face challenges in improving the quality of images displayed to users, particularly due to ghosting effects and resolution issues when performing temporal filtering, especially in low light conditions or with camera movement.
Innovation Solution
The system uses a camera with a sparse Bayer pattern sensor and IMU data to perform motion-compensated temporal filtering, generating high-resolution full-color output images by isolating and up-sampling red, green, blue, and monochrome pixel data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If temporal filtering is performed by combining multiple consecutively captured images, then image quality is improved, but ghosting effects occur when objects or camera move during capture
Solution Approach 1:
The patent segments the image processing into distinct color channels (red, green, blue) and a monochrome channel. Each channel is processed separately through temporal filtering, allowing independent optimization and reducing artifacts that would affect the composite image. The sparse Bayer pattern sensor physically segments pixel types across the sensor array, enabling this channel-separated processing approach.
Solution Approach 2:
The patent applies preliminary motion compensation using IMU data before performing temporal filtering. By predicting and correcting for camera motion in advance of the image combination process, the system prevents ghosting effects rather than attempting to correct them afterward. This preliminary alignment of images based on motion data ensures that corresponding pixels represent the same scene location.
2Illumination intensity
If multiple consecutively captured images are combined via temporal filtering, then low light image data is improved, but resolution and quality deteriorate due to camera movement and ghosting
Solution Approach 1:
The patent separates image processing into distinct color channels (R, G, B) and a monochrome channel, allowing each to be optimized independently. The monochrome channel, which has higher signal-to-noise ratio in low light, can be processed differently from the color channels, preserving resolution while improving overall image quality in low light conditions.
Solution Approach 2:
Motion compensation using IMU data is applied before temporal filtering to prevent degradation of resolution and quality. By pre-aligning images based on predicted motion, the system maintains sharp edges and fine details even when combining multiple low light images, avoiding the blur and ghosting that would otherwise occur.
3Loss of information
If a sparse Bayer pattern sensor is used to capture images, then color information is obtained, but resolution is reduced due to the sparse color pixel distribution
Solution Approach 1:
The patent processes each color channel (R, G, B) and the monochrome channel separately through temporal filtering and up-sampling. This channel-separated processing allows the sparse color pixels to be reconstructed more effectively by leveraging information from multiple frames, recovering resolution that would be lost in a single-frame processing approach.
Solution Approach 2:
Motion compensation is applied before up-sampling and combining channels. By establishing accurate geometric relationships between frames using IMU data, the system can interpolate missing color information more accurately during up-sampling, recovering resolution without introducing artifacts from misaligned pixel positions.
Data Source
AI summary
Techniques for generating a high resolution full color output image from lower resolution sparse color input images are disclosed. A camera generates images. The camera's sensor has a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and IMU data are fed into a motion model, which performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, a blue-only image, and a monochrome image. The color images are up-sampled to match the resolution of the monochrome image. A high resolution output color image is generated by combining the up-sampled images and the monochrome image.