Neural Network Inference Using Quantized Codebooks and Table Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face challenges due to a large number of parameters and computations, leading to high storage and energy consumption, necessitating a reduction in parameter and computation amounts.
Innovation Solution
The method involves quantizing input neurons and weights to identify similarities and distribution characteristics, using low-bit quantization and table lookup operations to reduce memory access and computation energy consumption, by grouping and clustering weights and neurons into codebooks for efficient representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network parameters and computation scale are increased to improve model performance, then accuracy and capability are improved, but storage capacity requirements and memory access energy consumption increase significantly
Solution Approach 1:
The patent creates codebooks that store pre-computed weight patterns and neuron activation patterns. During inference, instead of performing full matrix multiplications, the system copies and combines relevant codebook entries that match the input pattern, significantly reducing memory access requirements and energy consumption while maintaining computational accuracy.
Solution Approach 2:
The patent pre-computes and stores weight codebooks and neuron codebooks during a training phase. These pre-computed representations are stored in memory for rapid retrieval during inference, eliminating the need to perform complex computations repeatedly and reducing real-time memory access energy consumption.
2Reliability
If neural network parameters are increased to improve model capability, then accuracy is improved, but storage capacity requirements increase
Solution Approach 1:
Instead of storing all individual weight parameters, the patent stores compressed codebook representations that capture the essential weight patterns. These codebooks serve as compact copies that can be reused multiple times during inference, dramatically reducing storage requirements while preserving model capability.
Solution Approach 2:
The patent merges multiple weight parameters into consolidated codebook entries that represent groups of weights with similar characteristics. By combining redundant or similar weight patterns into single representative entries, the system reduces the total quantity of stored parameters while maintaining the computational power needed for accurate inference.
3Reliability
If large-scale computation is performed to improve neural network accuracy, then model performance is improved, but computation energy consumption increases
Solution Approach 1:
The patent replaces energy-intensive matrix multiplication operations with simpler codebook entry copying and combination operations. During inference, the system copies pre-computed codebook entries and combines them through simple arithmetic operations, avoiding the need for large-scale real-time computation and significantly reducing energy consumption while maintaining accuracy.
Solution Approach 2:
The patent performs complex computational work during the offline training phase to create optimized codebooks. During online inference, only simple lookup and combination operations are needed, shifting the computational burden to a preliminary stage where energy consumption is less critical and enabling low-energy operation during actual deployment.
Data Source
AI summary
The application provides an operation method and device. Quantized data is looked up to realize an operation, which simplifies the structure and reduces the computation energy consumption of the data, meanwhile, a plurality of operations are realized.