Neural Network Training With Binary-Ternary Backpropagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of deep neural networks is computationally intensive and time-consuming, and existing low-cost hardware solutions for on-chip learning face challenges in reducing complexity and latency, particularly in stochastic computing-based neural networks (SCNNs).
Innovation Solution
The method involves performing forward propagation using high-precision operations for conventional neural networks and binarized neural network parameters for SCNNs, and backward propagation using binary or ternary operations, estimating polarity and neutrality of gradients, and employing stochastic representations to enhance accuracy and reduce computational latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision operations are used for forward propagation in conventional neural networks, then accuracy is maintained, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the neural network operations into two distinct phases: forward propagation using high-precision operations to maintain accuracy, and backward propagation using simplified binary/ternary operations to reduce complexity. This segmentation allows each phase to use the appropriate precision level for its specific function.
Solution Approach 2:
The patent changes the precision parameter dynamically based on the propagation direction. During forward propagation, full-precision weights and activations are used to maintain accuracy. During backward propagation, weights and gradients are quantized to binary or ternary values, significantly reducing computational complexity while preserving training effectiveness.
2Measurement precision
If high-precision multipliers are used in neural network training, then computational accuracy is maintained, but area-complexity and cost increase
Solution Approach 1:
The patent replaces expensive high-precision multipliers with cheap binary/ternary operators for backward propagation. These simplified operators require significantly less hardware area and can be implemented using basic logic gates, making the system more cost-effective while maintaining sufficient accuracy through the two-phase approach.
Solution Approach 2:
The precision parameter is changed from high-precision floating-point to binary/ternary quantization specifically for the backward propagation phase. This parameter change reduces the hardware requirements from complex multipliers to simple logic operations, dramatically reducing area-complexity.
3Measurement precision
If conventional training methods are used for deep neural networks, then model accuracy can be achieved, but training time and processing resources are excessive
Solution Approach 1:
The training process is segmented into forward propagation (accuracy-critical) and backward propagation (computationally intensive). By applying different operational precision to each segment, the patent reduces overall training time while preserving the accuracy-dependent forward pass and optimizing the time-consuming backward pass.
Solution Approach 2:
The computational precision parameter is changed during backward propagation from standard floating-point to binary/ternary representation. This parameter change reduces the number of bits processed and simplifies arithmetic operations, directly reducing training time and processing resource requirements.
4Productivity
If binary or ternary operations are used for backward propagation, then computational efficiency is improved, but gradient estimation accuracy may be reduced
Solution Approach 1:
The patent incorporates feedback mechanisms where the simplified binary/ternary gradients from backward propagation are used to update weights, which then affect subsequent forward propagation accuracy. This feedback loop allows the system to adapt and maintain effectiveness despite the reduced precision in gradient estimation.
Solution Approach 2:
The precision parameter for gradient representation is changed to binary/ternary values during backward propagation. While this reduces gradient estimation accuracy compared to full-precision floating-point, it dramatically improves computational efficiency. The patent accepts this trade-off as the simplified operations enable practical implementation on resource-constrained devices.
Data Source
AI summary
There is provided a computer-implemented method of training a neural network, the neural network comprising a plurality of interconnected nodes arranged in a plurality of layers and having a plurality of weights associated therewith. The method comprises obtaining, at a computing device, a training dataset from a database, performing, at the computing device, a forward propagation on the training dataset to produce an output of the neural network, and performing, at the computing device, a backward propagation on the output of the neural network to update the plurality of weights, the backward propagation performed using an estimation of a polarity and a neutrality of gradients of the neural network.