Foveating Neural Network for Gaze-Adaptive Image Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing techniques using neural networks are computationally intensive, require large storage, and are time-consuming due to the need for separate networks to process different regions of an image, leading to inefficient processing and training.
Innovation Solution
An imaging system and method utilizing a single foveating neural network to perform demosaicking and image restoration on entire image data, identifying gaze and peripheral regions based on user gaze direction, reducing computational burden and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate neural networks are used for processing different regions of an image, then processing precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the image processing task into different regions (gaze region and non-gaze region) and applies different processing strategies to each region within a single neural network. The network divides the input image into multiple regions and processes them differently, achieving region-specific processing precision without requiring separate neural networks for each region.
Solution Approach 2:
The patent designs a single neural network that performs multiple functions: it processes both gaze and non-gaze regions, applies different processing intensities to different regions, and generates output images that emulate human visual characteristics. This multi-functional network eliminates the need for multiple separate networks while maintaining processing precision.
2Measurement precision
If multiple neural networks are used for image processing, then processing precision is improved, but productivity decreases
Solution Approach 1:
The patent merges the functionality of multiple separate neural networks into a single unified network. This single network processes both gaze and non-gaze regions simultaneously, eliminating the computational overhead of running multiple networks while maintaining the processing precision benefits of region-specific processing.
3Measurement precision
If separate neural networks are used for different image regions, then processing precision is improved, but loss of energy increases
Solution Approach 1:
The patent applies local quality by processing different regions of the image with different levels of detail and computational resources. The gaze region receives more intensive processing while non-gaze regions receive lighter processing, optimizing energy consumption by focusing computational effort only where visually critical.
4Measurement precision
If multiple neural networks are implemented, then processing precision is improved, but storage space increases
Solution Approach 1:
The patent implements a single universal neural network that handles all image processing tasks including gaze region processing, non-gaze region processing, and intermediate region processing. This eliminates the need to store multiple separate network models, reducing storage requirements while maintaining processing precision through region-specific processing within the unified network.
5Measurement precision
If multiple neural networks are used for image processing, then processing precision is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The patent segments the training process into different components corresponding to different image regions. The single neural network is trained with region-specific loss functions and processing parameters, allowing precise control over how different regions are processed while simplifying the overall training architecture compared to coordinating multiple separate networks.
Data Source
AI summary
Disclosed is an imaging system with an image sensor; and at least one processor configured to obtain image data read out by the image sensor; obtain information indicative of a gaze direction of a given user; and utilise at least one neural network to perform demosaicking on an entirety of the image data; identify a gaze region and a peripheral region of the image data, based on the gaze direction of the given user; and apply at least one image restoration technique to one of the gaze region and the peripheral region of the image data.