Neural Network Thresholding Logic With On-Demand Nonlinear Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant memory storage for monotonic quantized activations due to exponential growth in threshold values with bit-width, especially when using floating-point scale factors, which is inefficient and costly.
Innovation Solution
Implement thresholding logic that calculates non-linear, near-uniform thresholds on the fly using an initial value, step size, and residual errors, rather than storing them in memory, through techniques like sequential calculation or binary search trees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If monotonic quantized activations with floating-point scale factors are implemented using multi-thresholding operations in hardware, then accuracy benefits are preserved and expensive floating-point arithmetic is avoided, but memory storage cost for threshold values grows exponentially with the bit-width of the quantized activations
Solution Approach 1:
The patent transforms the threshold values from arbitrary non-linear values to a near-uniform parameterization where thresholds are generated by adding a constant step size to an initial threshold value. This parameter change reduces the storage requirement from exponential to linear, as only the initial threshold and step size need to be stored rather than all threshold values.
Solution Approach 2:
Instead of storing actual threshold values in memory, the patent creates a simplified copy or representation using just two parameters (initial threshold and step size) that can generate all necessary threshold values through calculation. This copying approach dramatically reduces memory storage while preserving the functional equivalence of the original thresholding operation.
2Reliability
If threshold values are stored explicitly in memory for quantization operations, then accurate thresholding is achieved, but hardware implementation becomes inefficient and costly due to high memory storage requirements
Solution Approach 1:
The patent changes the parameter representation from storing complete threshold value arrays to storing only the generating parameters (initial value and step size). This transformation maintains thresholding accuracy while dramatically improving hardware efficiency by reducing memory access operations and storage requirements.
Solution Approach 2:
The threshold values serve themselves by being generated on-demand through a simple calculation process rather than being retrieved from memory. The initial threshold and step size automatically generate all necessary threshold values through sequential addition, eliminating the need for explicit storage and improving hardware productivity.
Data Source
AI summary
Embodiments herein describe generating non-linear thresholds which can be used, for example, in a quantization operation of a neural network layer. In one example, a compiler can receive a set of non-linear thresholds (e.g., rounded integer thresholds) and determine delta values indicating respective differences between two thresholds of the non-linear thresholds. Because the thresholds are non-linear, the delta values are also different. The compiler can calculate residual errors to represent the differences between the deltas with respect to a step size. Using these residual errors, an initial threshold value, a step size between the thresholds, the non-linear thresholds can be calculated as needed, instead of having to store the non-linear thresholds in memory.