Neural Network Fixed Point Weight Conversion via Node Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face accuracy loss when converting between floating point and fixed point values during computations, as existing methods lack localized scaling values that could mitigate this conversion-induced loss.
Innovation Solution
The system determines localized scaling values for each neural network node based on the magnitude of its floating point weight values and the maximum representable value in fixed point arithmetic, converting floating point weight values to fixed point and vice versa to minimize accuracy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If floating point weight values are converted to fixed point using global scaling values, then computational efficiency is improved, but accuracy loss occurs during conversion
Solution Approach 1:
The patent applies local quality by determining individual scaling values for each node based on the magnitude of its floating point weight values. Instead of using a single global scaling value for the entire neural network, each node receives a customized scaling value that optimizes the fixed point representation for its specific weight range, thereby minimizing conversion accuracy loss while maintaining the computational efficiency benefits of fixed point arithmetic.
Solution Approach 2:
The patent changes the parameter of scaling values from a global constant to node-specific dynamic values. By computing individual scaling values based on the maximum absolute weight value of each node and the fixed point precision requirements, the system adapts the representation parameters to each node's specific needs, reducing quantization error and improving overall accuracy.
2Productivity
If fixed point arithmetic is used for neural network computations, then computational efficiency is improved, but precision loss occurs during numeric conversions
Solution Approach 1:
The system implements local quality by assigning individual scaling values to each node based on the magnitude of its floating point weight values. This localized approach ensures that each node's weight values are scaled optimally for fixed point representation, minimizing precision loss during conversion while preserving the computational efficiency advantages of fixed point arithmetic throughout the neural network.
Solution Approach 2:
The patent applies preliminary action by pre-computing individual scaling values for each node before the actual neural network computations begin. These scaling values are determined based on the maximum absolute weight values and the desired fixed point precision, allowing for accurate conversions to be performed in advance without affecting the speed of subsequent computations.
3Measurement precision
If node-specific scaling values are determined from floating point weight values, then conversion accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by computing the individual scaling values for all nodes before the actual neural network inference or training begins. The scaling values are determined once based on the floating point weight values and fixed point precision requirements, and then stored for use during computations. This upfront calculation eliminates the need for complex runtime adjustments, improving conversion accuracy while keeping the overall computational complexity manageable.
Data Source
AI summary
Methods, systems, and computer storage media for implementing neural networks in fixed point arithmetic computing systems. In one aspect, a method includes the actions of receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic; for each node of each layer of the neural network, determining a respective scaling value for the node from the respective set of floating point weight values for the node; and converting each floating point weight value of the node into a corresponding fixed point weight value using the respective scaling value for the node to generate a set of fixed point weight values for the node; and providing the sets of fixed point floating point weight values for the nodes to the processing system for use in processing inputs using the neural network.