Compressed CNN Weight Matrices for Multi-Core Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-core systems face storage space limitations and high computation requirements due to the large number of weights in sophisticated neural network models, particularly in edge devices used for image classification, which necessitates efficient compression and decompression techniques to maintain prediction accuracy without re-training.
Innovation Solution
A multi-core system employing quantization and matrix factorization to compress weight matrices into floating-point composites, followed by lossless encoding and decompression using a decompression matrix, allowing for efficient storage and execution of convolutional neural network models on edge devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sophisticated CNN models with large number of layers and dimensions are used for accurate image classification, then prediction accuracy is improved, but storage space requirement increases significantly
Solution Approach 1:
The weight matrix of each layer is segmented into multiple blocks, and each block is further divided into sub-blocks. This segmentation allows the large weight matrix to be stored in a compressed format where only essential components need to be stored, significantly reducing storage space while maintaining prediction accuracy.
Solution Approach 2:
A decompression matrix is introduced as an intermediary component. The original weight matrix is decomposed into a product of the decompression matrix and another matrix. The decompression matrix acts as a mediator that, when multiplied with the stored compressed weights, reconstructs the original weight matrix, enabling accurate inference with minimal storage.
2Measurement precision
If weights are stored in full precision floating-point format for accurate computation, then computation precision is improved, but storage space and processing overhead increase
Solution Approach 1:
The precision parameter of weight storage is changed from full floating-point format to a compressed format using integer or lower-precision representations. The decompression matrix compensates for this precision reduction, allowing the system to store weights with fewer bits while maintaining computation precision through the reconstruction process.
3Measurement precision
If more weights are stored for each layer to maintain model performance, then prediction accuracy is improved, but processing time and computational load increase
Solution Approach 1:
The weight matrices are pre-decomposed into compressed formats and decompression matrices during model training or preprocessing. This preliminary action allows the compressed weights to be stored efficiently, and during inference, the decompression can be performed quickly using optimized matrix multiplication, reducing processing time compared to storing and loading full-precision weights.
Data Source
AI summary
A system and method to compress application control data, such as weights for a layer of a convolutional neural network, is disclosed. A multi-core system for executing at least one layer of the convolutional neural network includes a storage device storing a compressed weight matrix of a set of weights of the at least one layer of the convolutional network and a decompression matrix. The compressed weight matrix is formed by matrix factorization and quantization of a floating point value of each weight to a floating point format. A decompression module is operable to obtain an approximation of the weight values by decompressing the compressed weight matrix through the decompression matrix. A plurality of cores executes the at least one layer of the convolutional neural network with the approximation of weight values to produce an inference output.