Residual Tensor Quantization for Accurate Low-Energy Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and deploying neural networks on general-purpose processors is computationally expensive, making real-time feature extraction challenging, and existing hardware accelerators face inefficiencies due to the lack of hardware floating-point support and high computational demands.
Innovation Solution
Implementing a neural network accelerator that converts input tensors from normal-precision floating-point to quantized-precision floating-point formats, using block floating-point formats to share exponents across multiple values, and selectively applying residual tensors based on error values and input metrics to enhance precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normal-precision floating-point format is used for neural network computations, then computational accuracy is maintained, but computational complexity and energy consumption increase significantly
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the precision format of floating-point numbers based on the exponent value. When the exponent falls within a predetermined range, block floating-point format is used to reduce energy consumption; otherwise, normal floating-point format is used to maintain accuracy. This dynamic parameter adjustment resolves the contradiction between accuracy and energy efficiency.
Solution Approach 2:
The system implements dynamics by making the precision format adaptive rather than static. The computational format switches between normal floating-point and block floating-point based on real-time exponent value assessment. This dynamic adaptation allows the system to optimize energy consumption without permanently sacrificing accuracy, resolving the contradiction between the two requirements.
2Measurement precision
If normal-precision floating-point format is used for neural network computations, then computational accuracy is maintained, but processing speed decreases
Solution Approach 1:
The patent changes the precision parameter dynamically based on exponent values. By using block floating-point format for exponents within a predetermined range, the system achieves faster processing speed while maintaining sufficient accuracy. This parameter change resolves the contradiction between accuracy and speed by optimizing the format selection.
Solution Approach 2:
The system employs dynamic format switching between normal and block floating-point representations. This dynamic approach enables the processor to select the most efficient format based on the current computational context, thereby improving processing speed without permanently compromising accuracy requirements.
3Use of energy by moving object
If quantized-precision floating-point format is used, then computational complexity and energy consumption are reduced, but precision is lost
Solution Approach 1:
The patent applies parameter changes by adjusting the precision level based on exponent values. When exponents are within a predetermined range, block floating-point format (quantized precision) is used to reduce energy consumption. For other cases, normal floating-point format is used to preserve precision. This conditional parameter adjustment resolves the contradiction between energy efficiency and precision.
4Device complexity
If hardware accelerators are implemented without floating-point support, then device complexity is reduced, but computational capability is limited
Solution Approach 1:
The patent resolves this contradiction by changing the numerical representation parameter from normal floating-point to block floating-point format. This parameter change enables the hardware accelerator to perform floating-point operations using simplified integer arithmetic logic, thereby maintaining computational capability while reducing hardware complexity and eliminating the need for dedicated floating-point support units.
Data Source
AI summary
Methods and apparatus are disclosed for providing emulation of quantized precision operations in a neural network. In some examples, the quantized precision operations are performed in a block floating-point format where values of a tensor share a common exponent. Techniques for selecting higher precision or lower precision can be used based on a variety of input metrics. When converting to a quantized tensor, a residual tensor is produced. In one embodiment, an error value associated with converting from a normal-precision floating point number to the quantized tensor is used to determine whether to use the residual tensor in a dot product calculation. Using the residual tensor increases the precision of an output from a node. Selection of whether to use the residual tensor can depend on various input metrics including the error value, the layer number, the exponent value, the layer type, etc.