Dense Shift Inner Product Operator for Low-Bit Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Low-bit neural networks, particularly low-bit shift neural networks, suffer from reduced accuracy when trained on large datasets and existing low-bit weight encodings do not optimally utilize bits, leading to sub-optimal performance in predicting new input data compared to full-precision networks.
Innovation Solution
The introduction of a dense shift inner product operator (IPO) and Sign-Sparse-Shift (S3) training methods, which encode weights using a dense shift encoding that includes a sign bit and multiple shift bits, allowing for more efficient computation and improved accuracy by re-parameterizing weights with continuous values for each bit of the encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If low-bit shift neural networks are used to reduce computing resources, then hardware efficiency and energy consumption improve, but prediction accuracy deteriorates
Solution Approach 1:
The patent changes the parameter representation by introducing dense shift encoding that re-parameterizes weights with continuous values for each bit of the encoding, allowing the model to achieve better accuracy with the same bit-width constraints while maintaining hardware efficiency
Solution Approach 2:
The patent adds a new dimension to the encoding by including multiple shift bits (b1, b2, b3) in addition to the sign bit, creating a multi-dimensional parameter space that enables more expressive weight representations within the same bit-width, thereby improving accuracy without increasing hardware complexity
2Device complexity
If existing low-bit weight encodings are used, then bit-width is reduced, but the number of value states is sub-optimal
Solution Approach 1:
The patent re-parameterizes the weight encoding by introducing continuous shift parameters that can take multiple discrete values, transforming the encoding from a simple sign-magnitude format to a dense shift format that achieves 2^n value states with n bits through combinatorial use of multiple shift bits
3Measurement precision
If dense shift encoding with continuous values is used, then accuracy improves, but computational complexity increases
Solution Approach 1:
The patent replaces traditional multiplication operations with shift operations in the forward pass, substituting computationally expensive mechanical multiplication with simpler bit-shifting operations that achieve the same effect through parameter re-parameterization, thereby reducing computational complexity while maintaining accuracy
Data Source
AI summary
Methods, systems and computer readable media using hardware-efficient bit-shift operations for computing the output of a low-bit neural network layer. A dense shift inner product operator (or dense shift IPO) using bit shifting in place of multiplication replaces the inner product operator that is conventionally used to compute the output of a neural network layer. Dense shift neural networks may have weights encoded using a low-bit dense shift encoding. A dedicated neural network accelerator is designed to compute the output of a dense shift neural network layer using dense shift IPOs. A Sign-Sparse-Shift (S3) training technique trains a low-bit neural network using dense shift IPOs or other bit shift operations in computing its outputs.