Hash-Based Weight Compression for DNN Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face memory and energy efficiency challenges due to increasing model size, leading to hardware under-utilization and memory bottlenecks, as existing weight compression techniques like quantization and pruning result in significant performance degradation or unstructured sparsity.
Innovation Solution
A hashing-based weight compression technique that groups random connections into buckets, sharing the same weight, allowing for a high compression ratio with negligible performance impact, using a reconfigurable hash table to construct a virtual weight matrix during training and inference, reducing memory footprint and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model size is increased to handle larger datasets, then processing capability is improved, but memory requirements increase
Solution Approach 1:
The patent segments the weight matrix into multiple smaller hash tables, where each hash table stores a portion of the weight values. This segmentation allows the model to handle large datasets by distributing weight storage across multiple manageable segments, effectively addressing the memory requirements issue while maintaining processing capability.
Solution Approach 2:
The patent changes the parameter of weight storage from dense matrix format to sparse hash table format. By using hash tables with configurable size and sparsity levels, the system can dynamically adjust memory usage while maintaining the ability to process large datasets, thus resolving the contradiction between processing capability and memory requirements.
2Quantity of substance
If weight compression is applied to reduce memory footprint, then memory efficiency is improved, but network performance degrades
Solution Approach 1:
The patent implements dynamic hash tables that can be reconfigured during training and inference. The hash table parameters (size, sparsity, bucket distribution) can be dynamically adjusted to optimize both memory footprint and network performance for different layers and operations, preventing performance degradation while achieving compression.
Solution Approach 2:
The patent incorporates feedback mechanisms where the performance impact of weight compression is monitored and used to adjust hash table parameters. This feedback loop ensures that compression ratios are optimized without causing significant network performance degradation, as the system adapts to maintain accuracy while reducing memory footprint.
3Productivity
If data transfer between external memory and accelerator is increased to support larger models, then model capacity is improved, but energy overhead increases
Solution Approach 1:
The patent merges the weight storage function with the compute function by implementing hash tables directly on the accelerator chip. This integration eliminates the need for frequent data transfer between external memory and accelerator, as weights can be generated and used in-place, significantly reducing energy overhead while maintaining model capacity.
Solution Approach 2:
The hash table structure enables the accelerator to generate weights on-demand during inference, eliminating the need for pre-loading entire weight matrices from external memory. This self-service approach allows the system to maintain large model capacity while minimizing external memory access and associated energy consumption.
4Speed
If I/O memory bottleneck is addressed by increasing memory bandwidth, then data availability is improved, but hardware under-utilization increases
Solution Approach 1:
The patent performs preliminary action by pre-computing hash table parameters and weight distributions during the training phase. This preliminary preparation allows the inference engine to operate with minimal memory access and computation, improving data availability speed while keeping the inference hardware simple and well-utilized, thus resolving the contradiction between data availability and hardware under-utilization.
Data Source
AI summary
Examples described herein relate to a neural network whose weights from a matrix are selected from a set of weights stored in a memory on-chip with a processing engine for generating multiply and carry operations. The number of weights in the set of weights stored in the memory can be less than a number of weights in the matrix thereby reducing an amount of memory used to store weights in a matrix. The weights in the memory can be generated in training using gradients from back propagation. Weights in the memory can be selected using a tabulation hash calculation on entries in a table.