CNN Weight Matrix Compression for Multi-Core Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, such as convolutional neural networks, require significant storage space and computation power for their weights, which is a challenge for edge devices with limited resources, necessitating efficient compression methods to maintain prediction accuracy without retraining.
Innovation Solution
A multi-core system compresses weight matrices through quantization and matrix factorization, followed by lossless encoding, allowing for decompression and approximation of weight values for inference, enabling efficient storage and computation on edge devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weights are stored in full precision floating-point format, then prediction accuracy is maintained, but storage space requirements increase significantly
Solution Approach 1:
The weight matrix is segmented into multiple blocks or tiles that can be processed and stored separately. This segmentation allows for selective precision storage where different blocks may use different precision levels, optimizing the balance between accuracy and storage space.
Solution Approach 2:
The precision parameter of weight storage is changed from full floating-point to lower precision formats (such as integer or reduced-precision floating-point). This parameter change reduces storage requirements while the system compensates through techniques like quantization awareness or calibration to maintain acceptable prediction accuracy.
2Reliability
If more weight values are used for each layer, then model complexity and accuracy improve, but storage space on edge devices is overwhelmed
Solution Approach 1:
Less critical weight values are extracted and removed from storage, keeping only the most important weights in full precision. The removed weights are either stored at lower precision or reconstructed on-demand, reducing overall storage requirements while maintaining model reliability for critical operations.
Solution Approach 2:
Instead of storing all weight values at full precision, the system applies partial precision storage where only essential weights maintain high precision. This partial action approach achieves acceptable model accuracy with significantly reduced storage space by being selective about which weights require full precision.
3Quantity of substance
If weights are compressed to reduce storage space, then storage requirements decrease, but computation time for decompression increases
Solution Approach 1:
Weights are pre-compressed into compact representations during model deployment or loading phase. The compression is performed in advance so that during inference, the system only needs to perform lightweight decompression or direct usage of pre-processed weight representations, minimizing real-time computation overhead.
4Measurement precision
If full precision weights are used, then inference accuracy is maintained, but processing efficiency on edge devices decreases
Solution Approach 1:
The precision parameter of weight representation is changed from full floating-point to lower precision formats. This parameter change reduces the computational burden on edge devices during inference, improving processing efficiency while the system maintains inference accuracy through careful quantization and calibration techniques.
Data Source
AI summary
A system and method to compress application control data, such as weights for a layer of a convolutional neural network, is disclosed. A multi-core system for executing at least one layer of the convolutional neural network includes a storage device storing a compressed weight matrix of a set of weights of the at least one layer of the convolutional network and a decompression matrix. The compressed weight matrix is formed by matrix factorization and quantization of a floating point value of each weight to a floating point format. A decompression module is operable to obtain an approximation of the weight values by decompressing the compressed weight matrix through the decompression matrix. A plurality of cores executes the at least one layer of the convolutional neural network with the approximation of weight values to produce an inference output.