Neural Network Derivation Using Correlation-Based Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks are susceptible to variations in parameters and input data, leading to incorrect inferred values and reduced accuracy, particularly when weight parameters are quantized, which can decrease the robustness of inference models.
Innovation Solution
A neural network derivation method that involves training a first neural network using a loss function and then adding a regularization term based on the correlation between latent features or inferred values of the first and second neural networks, to derive a second neural network with improved robustness to parameter and input data variations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural network is trained using a standard loss function for optimization, then the neural network achieves basic inference accuracy, but the neural network lacks robustness to variations in parameters and input data
Solution Approach 1:
The patent applies preliminary action by introducing a regularization term into the loss function during the training phase. This regularization term is constructed based on the correlation between latent features or inferred values of the neural network and its perturbed versions. By incorporating this preliminary regularization constraint during training, the neural network learns to maintain robustness against parameter and input data variations before actual inference occurs, thereby resolving the contradiction between achieving robustness and maintaining training simplicity.
2Productivity
If weight parameters are quantized to reduce computational complexity, then the inference model becomes more efficient, but the accuracy and robustness of the neural network decrease
Solution Approach 1:
The patent applies beforehand cushioning by preparing the neural network against quantization errors through the regularization term during training. The regularization term, which incorporates correlations with perturbed network versions, acts as a cushion that compensates for the accuracy loss that would otherwise occur when weight parameters are quantized. This allows the neural network to maintain both inference efficiency through quantization and acceptable accuracy by having been pre-adapted to parameter variations.
3Measurement precision
If the neural network is trained to achieve high accuracy on training data, then the model performs well on training examples, but the model becomes susceptible to adversarial attacks and input data variations
Solution Approach 1:
The patent applies feedback by incorporating the correlation between the neural network's latent features or inferred values and those of its perturbed versions into the regularization term. This creates a feedback mechanism during training where the network continuously adjusts its parameters to maintain consistency between the original and perturbed network outputs. This feedback loop ensures that the network learns robust features that are resistant to adversarial attacks and input variations while maintaining high training accuracy.
Data Source
AI summary
A neural network derivation method includes: (1) training a first neural network having a first parameter, using a first loss function for optimization; and (2) training the first neural network using a second loss function for optimization, after (1), the second loss function being obtained by adding a regularization term to the first loss function. After a second neural network having a second parameter obtained by adding a variation to the first parameter based on the first neural network is derived, the regularization term is determined based on a correlation between a latent feature of the first neural network and a latent feature of the second neural network or a correlation between an inferred value of the first neural network and an inferred value of the second neural network.