Compressed CNN Weight Matrices for Multi-Core Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, such as convolutional neural networks (CNNs), require significant storage space and computation power for inference tasks, which is a challenge for edge devices with limited resources.
Innovation Solution
A multi-core system that compresses weight matrices of CNN layers using quantization and matrix factorization, allowing for efficient storage and decompression of weights to maintain inference performance without retraining the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weights are stored in full precision floating-point format, then inference accuracy is maintained, but storage space requirements increase significantly
Solution Approach 1:
The weight matrix is segmented into multiple sub-matrices or blocks that can be processed and stored separately. This segmentation allows for selective precision storage where different blocks may use different compression strategies, balancing accuracy and storage efficiency.
Solution Approach 2:
The precision parameter of weight storage is changed from full floating-point to lower precision formats (e.g., integer quantization). This parameter change reduces storage requirements while the system compensates through techniques like calibration data or scaling factors to maintain acceptable inference accuracy.
2Measurement precision
If more layers and dimensions are added to CNN for complex image classification, then classification accuracy improves, but storage space and computation power requirements increase
Solution Approach 1:
The complex CNN model is segmented into multiple layers and operations that can be distributed across multiple cores. Each core handles specific layers or operations, allowing the complex model to run on edge devices with limited resources through parallel processing.
Solution Approach 2:
The problem is shifted from the model complexity dimension to the parallel processing dimension by distributing computations across multiple cores. This allows complex models to execute efficiently by leveraging temporal and spatial parallelism rather than increasing single-core computational power.
3Quantity of substance
If weights are compressed to reduce storage space, then storage requirements decrease, but inference performance may deteriorate
Solution Approach 1:
The system uses feedback mechanisms where compressed weights are evaluated on a validation set, and compression parameters are adjusted based on performance metrics. This feedback loop ensures that compression levels are optimized to maintain acceptable inference performance while minimizing storage requirements.
Solution Approach 2:
Compression parameters such as quantization bits and precision levels are dynamically adjusted based on the specific layer or weight importance. Critical weights maintain higher precision while less important weights use lower precision, optimizing the balance between storage efficiency and inference performance.
4Loss of time
If edge devices process data locally instead of transferring to cloud, then response time decreases and bandwidth usage reduces, but storage space becomes a critical limitation
Solution Approach 1:
The precision parameter of stored weights is changed to enable compact representation. By using quantized or compressed weight formats, edge devices can store sufficient model parameters locally without requiring excessive storage capacity, enabling fast local inference.
Solution Approach 2:
The model weights are segmented and prioritized based on importance, allowing edge devices to store only the most critical weights locally while potentially loading less critical weights on-demand or using approximation techniques, thus reducing storage requirements while maintaining acceptable performance.
Data Source
AI summary
A system and method to compress application control data, such as weights for a layer of a convolutional neural network, is disclosed. A multi-core system for executing at least one layer of the convolutional neural network includes a storage device storing a compressed weight matrix of a set of weights of the at least one layer of the convolutional network and a decompression matrix. The compressed weight matrix is formed by matrix factorization and quantization of a floating point value of each weight to a floating point format. A decompression module is operable to obtain an approximation of the weight values by decompressing the compressed weight matrix through the decompression matrix. A plurality of cores executes the at least one layer of the convolutional neural network with the approximation of weight values to produce an inference output.