Neural Network Output Calculation With Stochastic Rounded Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in reducing memory requirements, power consumption, and execution speed, particularly when implementing high-precision edge weights, which often result in significant degradation of classification accuracy when binarized for training.
Innovation Solution
The method involves generating discrete edge weights through stochastic rounding of high-precision weights and using these discrete weights to create multiple neural networks that are added together to produce a high-performance output, with intermediate results reused to accelerate calculations and reduce power consumption, while maintaining accuracy through targeted selection and buffering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision edge weights are used in neural networks, then classification accuracy is improved, but memory requirements and power consumption increase significantly
Solution Approach 1:
The patent segments the high-precision computation into multiple discrete-weight networks. Instead of using one high-precision network, multiple networks with discrete weights are generated and their outputs are combined, achieving high accuracy while using less memory since each individual network uses discrete weights.
Solution Approach 2:
The patent changes the weight parameter representation from continuous high-precision values to discrete values (e.g., binary or quantized levels). This parameter transformation reduces memory requirements while maintaining acceptable accuracy through the ensemble combination of multiple such networks.
2Measurement precision
If high-precision edge weights are used in neural networks, then classification accuracy is improved, but execution speed decreases
Solution Approach 1:
The patent divides the high-precision computation task into multiple discrete-weight network evaluations. Each individual network with discrete weights can be evaluated faster, and their combination produces the final high-accuracy result, thus improving overall execution speed while maintaining precision.
Solution Approach 2:
By transforming weights from continuous high-precision parameters to discrete parameters, the computational operations become simpler and faster (e.g., replacing multiplications with additions or lookups), thereby increasing execution speed while preserving accuracy through ensemble methods.
3Quantity of substance
If binarized edge weights are used for training, then memory requirements and power consumption are reduced, but classification accuracy degrades significantly
Solution Approach 1:
The patent merges multiple discrete-weight networks together by combining their outputs. Each individual network uses binarized weights for efficiency, but their collective output through addition or averaging achieves high classification accuracy, thus resolving the contradiction between memory efficiency and accuracy.
Solution Approach 2:
The patent uses multiple discrete-weight networks (excessive action) rather than a single one, where each network individually has limited accuracy due to discrete weights, but their combination provides the necessary accuracy while maintaining the memory efficiency benefits of discrete weights throughout.
Data Source
AI summary
A method for calculating an output of a neural network, including the steps of generating a first neural network that includes discrete edge weights from a neural network that includes precise edge weights by stochastic rounding; of generating a second neural network that includes discrete edge weights from the neural network that includes precise edge weights by stochastic rounding; and of calculating an output by adding together the output of the first neural network and of the second neural network.