Neural Network Fixed-Point Bit Allocation for Precision and Saturation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in finding a balance between memory saturation and precision loss when using fixed-point representations, leading to inefficiencies in processing speed and accuracy.
Innovation Solution
A method and system that analyze initial neural network parameters to adjust the size of integer and fractional portions of each layer, using a distribution or density function to optimize memory allocation based on a test input dataset, allowing for faster processing and improved precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the memory area allocated to the fractional portion is increased, then precision for small numbers is improved, but the risk of saturation for large numbers increases
Solution Approach 1:
The patent applies dynamics by making the allocation of bits between integer and fractional portions adaptive rather than fixed. The system dynamically adjusts the number of bits allocated to each portion based on the statistical distribution (mean and standard deviation) of the data in each layer, allowing the representation to optimize for either precision or saturation risk depending on the actual data characteristics of each layer
Solution Approach 2:
The patent changes the parameters of the fixed-point representation by calculating optimal bit allocations based on statistical parameters (mean and standard deviation) of the neural network data. This allows each layer to have customized integer and fractional bit allocations rather than using a uniform representation across all layers
2Reliability
If the memory area allocated to the integer portion is increased, then the risk of saturation is reduced, but precision for small numbers is lost
Solution Approach 1:
The system dynamically determines the optimal bit allocation for each layer based on the statistical properties of the data. By calculating the mean and standard deviation and using these to determine the number of bits needed for the integer portion, the system adapts to reduce saturation risk without unnecessarily sacrificing precision
Solution Approach 2:
The patent modifies the fixed-point representation parameters by setting the number of bits for integer and fractional portions based on calculated optimal values derived from data statistics, rather than using fixed allocation rules
3Measurement precision
If layer by layer processing is used to test all possible representations, then the right compromise between saturation and precision is found, but the process becomes tedious and slow
Solution Approach 1:
Instead of testing all possible representations, the patent directly calculates the optimal representation parameters (number of bits for integer and fractional portions) using statistical formulas based on the mean and standard deviation of the data. This analytical approach replaces the exhaustive testing methodology
Solution Approach 2:
The patent performs preliminary statistical analysis (calculating mean and standard deviation) to determine the optimal bit allocation before actual neural network execution. This preliminary calculation eliminates the need for iterative testing and trial-and-error processing
Data Source
AI summary
In accordance with an embodiment, a method includes reducing a size of at least one initial parameter of each layer of an initial multilayer neural network to obtain for each layer a set of new parameters defining a new neural network, wherein each new parameter of the set of new parameters has its data represented in two portions comprising an integer portion and a fractional portion; implementing the new neural network using a test input data set applied only once to each layer; determining a distribution function or a density function resulting from the set of new parameters for each layer; and based on the determined distribution function or density function, adjusting a size of a memory area allocated to the fractional portion and a size of the memory area allocated to the integer portion of each new parameter associated with each layer.