Neural Network Inference via Codebook Lookup Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) are computation-hungry workloads that stress limited hardware resources, particularly on mobile devices, and existing methods for efficient neural network inference require redesigning model structures or accelerators, which is resource-intensive and inefficient.
Innovation Solution
The solution involves determining centroids in codebooks for each layer of a neural network during training, using product quantization to precompute and store computation results in a lookup table, allowing for efficient table lookup-based inference by aggregating target computation results, thereby reducing computational load while maintaining model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional neural network computation is used, then model accuracy is maintained, but computational resources and energy consumption increase significantly
Solution Approach 1:
The patent precomputes the product of codebook centroids and weight matrices during the training phase, storing results in lookup tables. This preliminary computation eliminates the need for expensive matrix multiplications during inference, reducing computational resource consumption while maintaining model accuracy through efficient table lookup operations.
Solution Approach 2:
The patent creates compressed representations (codebooks) that copy essential feature information from the original data in a simplified format. By storing and manipulating these compact codebook representations instead of full-dimensional inputs, the system reduces computational load while preserving the information necessary for accurate predictions.
2Use of energy by moving object
If model structure redesign is performed to reduce computation, then resource usage decreases, but development complexity and time increase
Solution Approach 1:
The patent changes the computational parameters by transforming the standard neural network operation into a lookup-based operation. Instead of modifying the overall model architecture or creating custom hardware accelerators, the approach changes the computational method from matrix multiplication to table lookup, reducing resource usage without increasing development complexity.
3Productivity
If existing efficient inference methods are applied, then computation speed improves, but model accuracy deteriorates
Solution Approach 1:
By precomputing and storing the results of centroid-weight matrix products in lookup tables during training, the system enables fast inference through simple table lookups and aggregations. This preliminary action ensures that both speed (through efficient lookup) and accuracy (through learned centroid representations) are optimized simultaneously.
Solution Approach 2:
The patent incorporates a learning phase where the centroids and lookup tables are trained to minimize prediction error. This feedback mechanism ensures that the compressed representations in the lookup tables are optimized for accuracy while maintaining the computational efficiency of the lookup-based inference approach.
Data Source
AI summary
According to implementations of the subject matter described herein, a solution for neural network inference based on table lookup is provided. According to this solution, respective centroids in a first plurality of codebooks for a first layer of a neural network are determined along with a first weight matrix through a training procedure of the neural network. A first input for the first layer is divided into a first plurality of input sub-vectors, and target centroids are determined for the input sub-vectors based on respective distances between the input sub-vectors and the centroids. Target computation results of the target centroids with the first weight matrix are selected from a lookup table. A first output for the first layer is determined based on aggregation of the target computation results. In this way, better model accuracy can be achieved while leveraging the computation acceleration in table lookup-based model inference.