Quantized Sparse PCA Model Compression for Edge AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) consume significant computational and memory resources, making deployment on edge devices challenging due to resource constraints, and transferring weight data contributes to power consumption and latency during inference.
Innovation Solution
The method involves factorizing the original weight tensor into a codebook matrix and a latent matrix, where the matrix product approximates the original weight tensor, reducing the need for extensive weight data transfers and allowing for efficient computation of layer weights during inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deep neural networks are deployed on edge devices, then task performance is improved, but resource consumption (computational, power, memory) increases
Solution Approach 1:
The weight tensor is segmented into multiple codebooks, where each codebook contains a subset of weight vectors. This segmentation allows the network to process and transfer only relevant codebooks for each layer, reducing overall memory bandwidth requirements and power consumption while maintaining task performance on edge devices.
Solution Approach 2:
The patent extracts only the essential weight information needed for inference by representing weights as linear combinations of codebook vectors. This extraction reduces the amount of data that needs to be transferred from off-chip memory, thereby reducing power consumption and computational overhead during inference.
2Reliability
If weight data is transferred from off-chip to on-chip memory, then inference can proceed, but latency increases due to processing cores being idle
Solution Approach 1:
The codebooks are pre-computed and stored in a compressed format during the training phase. This preliminary action allows the inference phase to proceed more efficiently, as the processing cores can quickly retrieve and decompress only the necessary codebooks without waiting for extensive weight data transfers from off-chip memory, thereby reducing latency.
3Measurement precision
If full precision weight tensors are stored and transferred, then model accuracy is maintained, but memory and computational resources are consumed
Solution Approach 1:
The patent changes the representation parameters of weight tensors from full precision floating-point values to a compact format using codebook indices and coefficients. This parameter transformation significantly reduces the quantity of data stored and transferred in memory while maintaining model accuracy through the linear combination of codebook vectors.
Data Source
AI summary
A processor-implemented method includes retrieving, for a layer of a set of layers of an artificial neural network (ANN), a dense quantized matrix representing a codebook and a sparse quantized matrix representing linear coefficients. The dense quantized matrix and the sparse quantized matrix may be associated with a weight tensor of the layer. The processor-implemented method also includes determining, for the layer of the set of layers, the weight tensor based on a product of the dense quantized matrix and the sparse quantized matrix. The processor-implemented method further includes processing, at the layer, an input based on the weight tensor.