Camera-Specific Neural Network Training With Recaptured Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image signal processors (ISPs) optimized for visual image quality degrade information in images, leading to reduced inference accuracy in convolutional neural networks, as they are not designed to preserve details necessary for accurate image classification.
Innovation Solution
A method involving a control device that obtains an existing dataset, performs calibration operations, displays images on an electronic display, and generates a recaptured dataset by capturing images with a camera, training a neural network model on this dataset to enhance inference accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional ISP processes images to optimize visual image quality, then visual image quality is improved, but inference accuracy deteriorates due to information removal
Solution Approach 1:
The patent creates a recaptured dataset by capturing images of displayed images, producing copies of training images in the exact raw format that the camera produces during inference. This copying approach allows the neural network to be trained on data that perfectly matches the inference input format, eliminating the information loss problem that occurs when traditional ISP processing is applied.
Solution Approach 2:
The patent changes the format parameter of the training data from processed sRGB images to raw camera output format. By displaying images and recapturing them in raw format, the system transforms the training data parameters to match the inference input parameters, ensuring that the neural network learns from data with the same characteristics as real inference inputs.
2Measurement precision
If ISP removes noise and artifacts to improve visual quality, then visual image quality is improved, but inference accuracy deteriorates
Solution Approach 1:
By copying training images through the complete camera pipeline including the ISP, the patent ensures that the neural network is trained on data that includes all the same noise characteristics, artifacts, and processing effects that will be present during inference. This eliminates the problem of training on cleaned data while inferring on noisy data.
Solution Approach 2:
The system uses the camera's own ISP to process the recaptured images, creating a feedback loop where the same processing pipeline that operates during inference is also used during training data generation. This feedback mechanism ensures consistency between training and inference conditions.
3Ease of manufacture
If datasets are acquired in standard formats like sRGB, then ease of manufacture is improved, but adaptability to specific camera formats deteriorates
Solution Approach 1:
Instead of converting raw camera images to sRGB format (the traditional approach), the patent inverts the process by displaying sRGB images and recapturing them in raw format. This inversion allows the system to start with easily available standard format images and transform them into the specific camera format needed for training, solving both the ease of manufacture and adaptability problems.
Solution Approach 2:
The patent uses copying to transform data from one format to another by capturing images of displayed images. This copying mechanism through the camera pipeline automatically adapts the training data to the specific camera format without requiring complex conversion algorithms or losing information.
Data Source
AI summary
The system and methods of the various embodiments enable convolutional neural networks to be trained for improved accuracy for a particular class of cameras, where the training is performed on one or more pre-existing datasets by recapturing the existing datasets using one or more cameras of the same class. Further, the system and methods of the various embodiments enable the accuracy of a convolutional neural network to be evaluated after it has been deployed on a target device, using a hardware-in-the-loop approach to validation.