Neural Network Training Dataset Generation for Meta-Prism Image Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for generating training datasets for neural network learning models are inadequate for image sensors with meta-prism arrays, as they require complex calculations to account for parallax between pixels, leading to inefficiencies in data processing and performance.
Innovation Solution
A method is developed to generate a training dataset by receiving an open images dataset, creating a target image through parallax-based sampling and sub-pixel shift averaging, and generating an input image using a point spread function and quad Bayer pattern filtering, which compensates for the image sensor's meta-prism and pixel array characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If demosaicing processing is performed on image sensor data with meta-prism arrays, then color information can be reconstructed, but parallax restoration processing becomes required due to parallax between pixels
Solution Approach 1:
The patent applies preliminary action by generating parallax-restored training images in advance using high-resolution reference images and point spread functions. This pre-processing creates a dataset that already contains the parallax correction information, so the neural network learns to map raw sensor data to parallax-corrected results without requiring complex runtime parallax restoration processing.
Solution Approach 2:
The patent uses copying by creating synthetic training datasets from reference images. High-resolution reference images are processed through simulated point spread functions to generate corresponding parallax-restored images, creating paired training data (raw image, restored image) that teaches the neural network the parallax correction transformation without requiring actual complex processing during inference.
2Reliability
If neural network learning is performed with large amounts of missing data, then good performance can be achieved, but complex calculation processes are required
Solution Approach 1:
The patent performs preliminary action by pre-generating complete training datasets with parallax restoration already applied. Instead of requiring the neural network to handle complex parallax calculations during training, the training data is prepared in advance with the correct parallax-restored images, allowing the network to learn the transformation pattern more efficiently with simpler calculations.
Solution Approach 2:
The patent introduces an intermediary element - the point spread function - that mediates between the raw sensor data and the final parallax-restored image. The PSF serves as a mathematical model that captures the parallax effect, allowing the neural network to learn a standardized transformation rather than dealing with raw complex geometric relationships between pixels.
3Productivity
If standard image processing methods are used for meta-prism sensors, then processing is simpler, but resolution improvement is limited
Solution Approach 1:
The patent applies preliminary action by pre-calculating point spread functions and generating parallax-restored reference images before training. This allows the neural network to learn from high-quality training data that incorporates resolution enhancement, enabling the model to achieve better resolution performance while maintaining efficient processing during actual use.
Solution Approach 2:
The patent uses parameter changes by applying point spread functions with specific mathematical parameters to transform reference images. By varying the PSF parameters and generating multiple training samples with different transformations, the neural network learns to handle the parallax effect and enhance resolution through learned parameter adjustments rather than simple interpolation.
Data Source
AI summary
A method of generating a training dataset of a neural network learning model of an electronic device includes: receiving an open images dataset; generating a target image from the open images dataset; generating an input image from the open images dataset; and generating a training dataset of a neural network learning model for compensating for an image captured by an image sensor that includes a meta-prism and a pixel array, the training dataset including the target image and the input image.