Compressed Neural Networks Using Sparse Tensor Encoding for Fast Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks require significant storage space and transmission time due to their large size, and existing compression methods do not effectively address the need for increased inference speed and reduced power consumption.
Innovation Solution
A computer-implemented method for generating a compressed neural network by reordering the weight tensor into a 2D sparse bitmap, dividing it into groups of coefficients, selecting an encoding mode, and encoding the quantized reordered weight tensor to form a representation of the compressed neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the neural network is compressed to reduce storage space and transmission time, then the storage and transmission requirements are reduced, but the inference speed may decrease and power consumption may increase
Solution Approach 1:
The weight tensor is divided into multiple non-overlapping groups of coefficients (GOCs), where each GOC is processed and encoded independently. This segmentation allows for efficient parallel processing during inference, maintaining high inference speed while enabling comprehensive compression of the entire weight tensor through systematic grouping and encoding of all weight coefficients.
2Quantity of substance
If the neural network is compressed to reduce storage space, then the storage requirements are reduced, but the transmission time may increase due to complex compression and decompression processes
Solution Approach 1:
The weight tensor is reordered into a 2D sparse bitmap format and divided into GOCs before the actual compression encoding process. This preliminary reordering and grouping structure optimizes the subsequent encoding operation, enabling more efficient compression that reduces both storage space and transmission time by minimizing the computational overhead during the encoding phase.
3Quantity of substance
If the neural network is compressed to reduce storage space and transmission time, then the storage and transmission requirements are reduced, but the power consumption during inference may increase
Solution Approach 1:
The encoding mode is dynamically selected based on rate-distortion measurements, adjusting the compression parameters to optimize the balance between compression ratio and inference efficiency. This adaptive parameter adjustment ensures that the compressed representation maintains optimal performance characteristics, reducing power consumption during inference by selecting encoding configurations that minimize computational overhead while preserving necessary accuracy.
Data Source
AI summary
The disclosure relates to technology for generating a compressed neural network. A weight tensor is received from a neural network to be compressed, and it is reordered to be compressed to have an inner two-dimensional (2D) shape and a 2D sparse bitmap. A layered structure is generated that represents the reordered weight tensor, and the reordered weight tensor is divided into a group of coefficients (GOCs). An encoding mode is selected to generate a quantized reordered weight tensor using one of a codebook or direct quantization, and a column swapped quantized reordered weigh tensor is generated. A compressed neural network is formed by encoding and the compressed representation of the neural network is transmitted to a target system for decompression.