Stochastic Outer Product Computation for Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks is computationally intensive and resource-demanding, particularly due to the need for extensive outer product computations, which hinders their further application.
Innovation Solution
An electronic system and method for computing outer product matrix items using stochastic representation of real numbers, involving bitwise AND operations and data formatting to reduce resource requirements and improve accuracy, enabling efficient computation of outer products for deep neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional digital computation methods are used for outer product computations in deep neural network training, then computational accuracy is maintained, but computational resource demands and training time increase significantly
Solution Approach 1:
The patent replaces conventional digital multiplication operations with stochastic computing operations. Real numbers are converted to stochastic bit streams where the probability of a bit being 1 corresponds to the magnitude of the real number. Outer product computation is performed by generating stochastic representations of input vectors and computing their element-wise products through probabilistic operations, significantly reducing computational complexity and resource requirements while maintaining acceptable accuracy for neural network training applications
2Productivity
If stochastic representation is used to reduce computational resources, then resource efficiency improves, but computational accuracy may deteriorate
Solution Approach 1:
The patent employs parameter optimization in the stochastic computation process. Key parameters including the length of stochastic bit streams, the number of samples used for computing outer products, and the precision of stochastic-to-deterministic conversion are carefully tuned. By optimizing these parameters, the system achieves a balance where computational efficiency is dramatically improved while accuracy remains sufficient for effective deep neural network training
Solution Approach 2:
The patent incorporates feedback mechanisms to monitor and adjust the precision of stochastic computations. By evaluating the impact of stochastic approximations on training outcomes and adjusting computational parameters accordingly, the system maintains accuracy while benefiting from reduced resource consumption. This feedback-driven approach allows dynamic adaptation of computation precision based on actual training needs
Data Source
AI summary
The present disclosure relates to an electronic system for computing items of an outer product matrix, for each item of at least part of the items of the matrix. The system is configured to receive a pair of real numbers of two vectors, the pair corresponding to the item. The system is further configured to compute a stochastic representation of the real numbers resulting in two sets of bits, the set of bits comprising a subset of bits representing the real number and a sign bit indicative of the sign of the real number. The system is further configured to perform a sequence of digital operations using the two sets of bits to provide a representation of the item.