Image Sensor Signal Processing Using Selective Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing portable electronic devices and image capturing devices equipped with image sensors face challenges such as high power consumption, large size, and heavy weight, while also struggling with low performance in image processing.
Innovation Solution
A processor-implemented method that involves obtaining a color filter array (CFA) input image, preprocessing it based on pattern information, generating an inferred image using an artificial neural network (ANN) model, and creating an output image by selecting pixel values from the preprocessed and inferred images, optimized for each color type of the CFA.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional image processing methods are used in portable electronic devices, then device size and power consumption are reduced, but image processing performance deteriorates
Solution Approach 1:
The image processing task is segmented into two distinct stages: a lightweight preprocessing stage that performs color space conversion and basic filtering, and a selective neural network inference stage that processes only specific pixel locations. This segmentation allows the system to achieve high processing performance where needed while maintaining low power consumption for the majority of pixels.
Solution Approach 2:
Instead of applying computationally intensive neural network processing to all pixels, the method applies full ANN inference only to specific pixel locations (e.g., green pixels in Bayer pattern) that benefit most from it. Other pixels use the lighter preprocessing results, creating a partial action approach that optimizes the balance between performance and power consumption.
2Reliability
If high-performance image processing is implemented, then image quality is improved, but device complexity increases
Solution Approach 1:
The processing pipeline is divided into a simple preprocessing component with well-defined mathematical operations and a selective neural network component. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining high image quality through the neural network's selective application.
Solution Approach 2:
The preprocessing step performs color space conversion and basic image processing operations before neural network inference. This preliminary action prepares the data in an optimal format for the ANN, reducing the computational burden on the neural network and simplifying the overall processing architecture.
3Measurement precision
If comprehensive neural network processing is applied to all pixels, then demosaicing accuracy is improved, but computational complexity increases
Solution Approach 1:
Different processing qualities are applied to different pixel locations based on their specific needs. Pixels at certain locations (e.g., green pixels in Bayer pattern) receive full neural network processing for maximum demosaicing accuracy, while other pixels use the lighter preprocessing results. This local quality approach optimizes demosaicing accuracy where it matters most while reducing overall computational complexity.
Solution Approach 2:
The neural network is applied partially to only those pixel locations that require its full processing capability. This partial application maintains high demosaicing accuracy for critical pixels while significantly reducing the total computational burden compared to applying the network to all pixels.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A processor-implemented method includes obtaining a color filter array (CFA) input image, obtaining pattern information corresponding to a CFA, preprocessing the input image based on the pattern information, generating an inferred image by inputting the preprocessed input image to an artificial neural network (ANN) model, and generating an output image by selecting a pixel value, for each pixel, from one of the preprocessed input image and the inferred image, based on the pattern information.