Residual Quantization of Bit-Shift Weights in Neural Network Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require large memory spaces to store weights, which scales with their complexity, and existing neural network accelerators inefficiently utilize Look-Up Table (LUT) resources for Multiply-Accumulate (MAC) operations, leading to high hardware usage costs and memory inefficiencies.
Innovation Solution
Encoding 16-bit weights as bit shifts using residual quantization, reducing the number of shifts from 16 to 3, and utilizing LUT resources for these operations, thereby reducing memory requirements and improving computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If 16-bit weights are stored in full precision, then manufacturing precision is maintained, but device complexity and memory size increase
Solution Approach 1:
The patent changes the parameter representation from full 16-bit weights to a compact format using bit-shift values. Each weight is represented by specifying a base value and a small number of bit-shift adjustments (typically 3 shifts), reducing the memory required per weight from 16 bits to a much smaller representation while maintaining computational accuracy through the shift operations that reconstruct the full precision when needed.
2Measurement precision
If traditional MAC operations are used, then computational accuracy is maintained, but device complexity and hardware resource usage increase
Solution Approach 1:
The patent extracts the multiplication operation from the traditional MAC (Multiply-Accumulate) unit and replaces it with bit-shift operations. By taking out the multiplication function and substituting it with shifts and adds, the design eliminates the need for complex multipliers in the MAC unit, reducing hardware resource usage while maintaining computational accuracy through the mathematical equivalence of the operations.
3Device complexity
If LUT resources are used for MAC operations, then device complexity is reduced, but manufacturing precision may deteriorate
Solution Approach 1:
The patent introduces bit-shift operations as an intermediary mechanism between the LUT resources and the MAC operations. The LUTs store pre-computed bit-shift values that serve as mediators, enabling the system to perform accurate weight applications using simple shift and add operations instead of requiring complex multipliers, thus maintaining precision while reducing hardware complexity.
Data Source
AI summary
A neural network accelerator reads encoded weights from memory. All 1 bits in a weight except the first three are discarded. The first three leading 1 bits in the weight are encoded as three bit-shift values to form the encoded weight. The three bit-shift values are applied to a bit shifter to shift a node input to obtain three shifted inputs that are accumulated to generate the node output. Node complexity is reduced since only 3 shifts are performed rather than up to 15 shifts for a 16-bit weight. The bit shifter and accumulator for a node can be implemented by Look-Up Tables (LUTs) without requiring a Multiply-Accumulate (MAC) cell in a Field-Programmable Gate Array (FPGA). Quantization bias is reduced using a histogram analyzer that determines a weighted average for each interval between quantized weights. The third bit-shift value is incremented for weights in the interval above the weighted average.