Sparsity-Aware Compression for Neural Network Weight Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with highly sparse weights and activations face inefficiencies in loading times, leading to compute phase bottlenecks due to the presence of numerous zeros, which hinder processing speed and efficiency.
Innovation Solution
A sparsity-aware compression scheme that encodes sparse weights and skips loading of sparse activations, using a bitmap representation to efficiently distribute non-zero values across processing elements, allowing for faster decompression and reduced latency by leveraging sparsity to accelerate data loading and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sparse weights and activations are loaded in conventional format, then data completeness is maintained, but load time increases and processing efficiency decreases
Solution Approach 1:
The patent extracts and removes zero-valued weights and activations from the data stream, loading only non-zero elements into the processing elements. This is achieved by identifying the positions and values of non-zero elements separately from zero elements, thereby reducing the total data volume that must be loaded and processed, directly addressing the load time and processing efficiency contradiction
Solution Approach 2:
Instead of loading all weights and activations and then processing zeros during computation, the patent inverts the approach by pre-identifying and loading only non-zero elements. The processing elements are configured to handle the sparse data structure natively, eliminating the need to load and process zero values, thus resolving the efficiency-time tradeoff
2Speed
If all weights are loaded into processing elements, then computation can proceed, but memory bandwidth is consumed and load-induced stalls occur
Solution Approach 1:
The patent extracts only the necessary non-zero weight values and their positions from the full weight matrix, loading minimal data into the processing elements. This extraction approach reduces memory bandwidth consumption while maintaining computation speed by ensuring that only useful data is transferred from memory to processing units
Solution Approach 2:
The patent performs preliminary identification and organization of non-zero weight elements before the actual computation phase. By pre-processing the weight data to separate non-zero elements and their positions, the system avoids load-induced stalls during computation, as the processing elements receive ready-to-use sparse data structures
3Quantity of substance
If sparse data is compressed and stored, then storage efficiency improves, but decompression complexity increases
Solution Approach 1:
The patent segments the sparse data into distinct components: non-zero weight values and their corresponding positions. This segmentation allows for efficient compression by storing only the essential non-zero elements separately from the zero elements, reducing data volume while keeping the decompression process relatively simple through structured organization
Solution Approach 2:
The patent changes the representation parameters of sparse data by using coordinate-based indexing and value storage instead of dense array format. This parameter transformation enables compact storage of sparse weights by recording only (position, value) pairs for non-zero elements, achieving compression without requiring complex decompression algorithms
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems, apparatuses and methods may provide for technology that identify an assignment of weights of a workload to a plurality of processing elements, where the workload is to be associated with a neural network. The technology generates a representation that is to represent whether each of the weights is a zero value or a non-zero value. The technology further stores the representation into partitions of a storage structure based on the assignment of the weights, where the partitions are each to be dedicated to a different one of the processing elements.