Conditioned Neural Network Weights to Prevent Accumulator Overflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based image and video coding technologies face challenges in achieving device interoperability and bitexact processing results across different platforms, particularly in entropy model-based coding and compressing/decompressing of data.
Innovation Solution
A method is introduced to condition neural network weights by defining integer lower and upper thresholds to avoid accumulator register overflow, ensuring integer-valued weights that maintain bitexact reproductions across devices, using techniques like clipping, scaling, and rounding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard neural network training methods are used, then the model can learn from training data, but the model memorizes training examples and performs poorly on out-of-distribution data
Solution Approach 1:
The patent applies parameter changes by modifying the weight values of neural network parameters based on their sensitivity to input perturbations. Weights are scaled down according to their sensitivity metrics, transforming the original weight distribution to reduce overfitting and improve generalization to out-of-distribution data.
Solution Approach 2:
The patent performs preliminary action by computing sensitivity metrics and determining weight scaling factors before the actual training process. This pre-computation of sensitivity-based weight adjustments prepares the model to better handle out-of-distribution data from the outset, rather than attempting to correct overfitting during training.
2Reliability
If more training data is collected to improve generalization, then out-of-distribution performance may improve, but data collection and processing time increases
Solution Approach 1:
Instead of collecting more data, the patent changes the weight parameters of the neural network to achieve better generalization. This parameter transformation approach provides an alternative pathway to improve reliability without incurring the time costs of additional data collection and processing.
3Productivity
If the neural network architecture is simplified to reduce computation, then training speed increases, but model expressiveness and accuracy decrease
Solution Approach 1:
The patent changes the values and distribution of weight parameters rather than modifying the network architecture. This allows the model to maintain its full expressiveness and accuracy capability while the parameter transformation itself is computed efficiently, avoiding the need to simplify the architecture.
4Reliability
If weight regularization is applied to prevent overfitting, then generalization improves, but model capacity to learn complex patterns decreases
Solution Approach 1:
The patent applies parameter changes by selectively scaling weights based on their sensitivity to perturbations rather than applying uniform regularization. This differentiated approach preserves the capacity to learn complex patterns in directions that are less sensitive to input variations while reducing overfitting in sensitive directions, thereby maintaining model versatility.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
The present disclosure relates to a method of operating a neural network based on conditioned weights. The method comprises defining integer lower and upper threshold values for values of integer numbers comprised in data entities of input data for the neural network layer. If a value of an integer numbers comprised in a data entity of the input data is smaller than the lower threshold value, the value of the integer number comprised in the data entity of the input data is clipped to the lower threshold value, and if a value of an integer number comprised in a data entity of the input data is larger than the upper threshold value, the value of the integer number comprised in the data entity of the input data is clipped to the upper threshold value and integer valued weights are determined based on the lower threshold value, the upper threshold value and a pre-defined accumulator register size, such that integer overflow of the accumulator register can be avoided.