Neural Demosaicking of Subsampled RAW Pixels for High Frame Rates

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing image processing technologies face limitations in processing high-resolution images at high frame rates due to computational overhead, requiring excessive computing power and failing to meet visual quality requirements for immersive extended-reality applications, often resulting in poor viewing experiences.

Innovation Solution

An imaging system and method utilizing neural networks for interpolation and demosaicking of subsampled pixels, reducing computational burden by selectively reading and processing RAW image data with a subsampling mask, followed by neural network-based interpolation and demosaicking to generate high-quality image frames.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If full pixel readout is used to process high-resolution images, then image quality is improved, but processing time and computational resources increase significantly

Engineering Contradiction:

Improveimage qualityVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent segments the image processing task into two stages: first, a reduced set of pixels is read out and processed quickly to generate a low-resolution intermediate image; second, this intermediate image is used to reconstruct the full high-resolution image. This segmentation allows the system to achieve high image quality without the time penalty of processing all pixels at full resolution simultaneously.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary processing by reading out and processing a subset of pixels first to create an intermediate image. This preliminary action provides enough information to guide the subsequent reconstruction of the full-resolution image, thereby reducing the overall processing time while maintaining image quality.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If full pixel readout is used to achieve high resolution, then image quality is improved, but the total number of pixels that can be processed at a given frame rate is limited

Engineering Contradiction:

Improveimage resolutionVSAvoidframe rate

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent divides the pixel readout process into two phases: a first phase where a reduced set of pixels is read out at high speed to maintain frame rate, and a second phase where the remaining pixels are processed using the intermediate image. This segmentation enables the system to support higher resolutions at the same frame rate by not requiring all pixels to be read out simultaneously.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by processing only a subset of pixels in the first readout phase rather than all pixels. This partial processing is sufficient to generate an intermediate image that can then be used to reconstruct the full-resolution image, thereby increasing the total number of pixels that can be handled at a given frame rate.

Inventive Principle:

Principle #16Partial or excessive action

3Area of stationary object

If multiple cameras per eye are used to obtain large field of view at high frame rate, then field of view is improved, but system complexity and processing requirements increase

Engineering Contradiction:

Improvefield of viewVSAvoidnumber of cameras

Core Design Contradiction:

Area of stationary objectVSDevice complexity

Solution Approach 1:

The patent makes a single image sensor perform multiple functions: it captures both the reduced set of pixels for high-speed processing and the full-resolution data for high-quality reconstruction. This multi-functionality eliminates the need for multiple cameras while achieving the same field of view and frame rate performance.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Loss of energy

If output data rate from image sensor is reduced to decrease bandwidth requirements, then bandwidth consumption is reduced, but image resolution deteriorates

Engineering Contradiction:

Improvebandwidth consumptionVSAvoidimage resolution

Core Design Contradiction:

Loss of energyVSMeasurement precision

Solution Approach 1:

The patent creates an intermediate image copy from the reduced set of readout pixels. This intermediate image serves as a template that is then used to reconstruct the full-resolution image data. By copying and utilizing this intermediate representation, the system can reduce bandwidth consumption during transmission while recovering the full image resolution through computational reconstruction.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS12530739B2Interpolating and demosaicking subsampled pixels using neural networks

Publication Date: 2026.01.20 VARJO TECH OY

AI summary

Disclosed is imaging system having an image sensor with pixels arranged on photo-sensitive surface. The image sensor is employed to read out RAW image data, wherein at least a portion of RAW image data is subsampled during read out from at least a portion of photo-sensitive surface. Processor(s) are configured to obtain RAW image data read out by image sensor; and perform interpolation and demosaicking on the portion of RAW image data using neural network(s), to generate first intermediate image data, wherein an input of neural network(s) is the portion of RAW image data and subsampling mask(s), wherein subsampling mask(s) indicates pixels that have not been read out.