Hybrid Compression Scheme for Neuromorphic Synaptic Weight Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neuromorphic hardware faces challenges in storing synaptic weights due to limited memory capacity, as existing compression schemes are inefficient for architectures with sparse synaptic connections and varying weight precisions.
Innovation Solution
A hybrid compression scheme combining run-length encoding (RLE) and dictionary encoding, along with a compressionless approach, is implemented to efficiently store synaptic weights by identifying and encoding non-zero values, zero runs, and commonly occurring weights, using a pointer table and dictionary memory to reduce memory size without degrading recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression schemes are used to store synaptic weights, then memory capacity requirements are reduced, but compression efficiency is insufficient for sparse synaptic connections with varying weight precisions
Solution Approach 1:
The patent segments the synaptic weight storage into multiple precision levels (e.g., 1-bit, 2-bit, 4-bit, 8-bit precision groups). Different segmentation strategies are applied based on the sparsity pattern and weight distribution, allowing efficient compression for each segment while adapting to the specific characteristics of sparse neural network weights.
Solution Approach 2:
The patent applies different compression techniques to different regions of the weight matrix based on local characteristics. Specifically, run-length encoding is applied to sparse regions with long sequences of zeros, while dictionary encoding is applied to regions with recurring weight patterns. This local adaptation maximizes compression efficiency for each specific region's properties.
2Quantity of substance
If run-length encoding is applied to compress zero runs in synaptic weights, then memory usage is reduced for sparse connections, but additional overhead bits are required to encode run lengths
Solution Approach 1:
The patent applies run-length encoding selectively only to regions where zero runs exceed a threshold length, rather than uniformly to all weights. This partial application avoids the overhead of encoding short runs that would consume more bits than the original representation, thereby reducing overall complexity while maintaining compression benefits for significant sparse regions.
3Quantity of substance
If dictionary encoding is used to compress commonly occurring weights, then memory capacity is reduced, but dictionary construction and lookup complexity increases
Solution Approach 1:
The patent pre-computes and stores the dictionary of common weight patterns during an offline training phase or initialization period. This preliminary action allows the runtime system to simply perform lookups without complex construction operations, significantly reducing the operational complexity while maintaining the memory compression benefits of dictionary encoding.
4Measurement precision
If higher precision is used to store synaptic weights, then recognition accuracy is maintained, but memory capacity requirements increase significantly
Solution Approach 1:
The patent implements dynamic precision allocation where different synaptic weights are stored at different precision levels based on their importance and contribution to network accuracy. Critical weights maintaining recognition accuracy are stored at higher precision (e.g., 8-bit), while less critical weights are stored at lower precision (e.g., 1-bit or 2-bit). This dynamic approach maintains overall recognition accuracy while significantly reducing total memory capacity requirements.
Data Source
AI summary
Systems, apparatuses and methods may provide a hybrid compression scheme to store synaptic weights in neuromorphic cores. The hybrid compression scheme utilizes a run-length encoding (RLE) compression approach, a dictionary-based encode compression scheme, and a compressionless encoding scheme to store the weights for valid synaptic connections in a synaptic weight memory.