Neural Network Weight Masking for Sparse Kernel Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face challenges in efficiently storing and processing weights due to limited space in internal memory buffers, leading to high computational costs and memory overspill issues, especially when dealing with large weight tensors containing many zero-value weights.
Innovation Solution
The method involves generating mask bits to identify zero-value weights within a kernel of weights, allowing for the storage and transmission of only non-zero value weights and mask bits, which reduces memory requirements and bandwidth needed for storage and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all weights are stored in the weight buffer, then complete weight information is available for processing, but memory space is excessively large and costly
Solution Approach 1:
The patent extracts only the non-zero weight values from the complete weight set and stores them separately in the weight buffer, while using mask bits to indicate the positions of these non-zero weights. This extraction principle reduces the quantity of data stored while maintaining the ability to reconstruct the complete weight information when needed.
Solution Approach 2:
The patent applies different storage strategies to different parts of the weight data: non-zero weights are stored with their actual values, while zero weights are represented only by mask bits indicating their positions. This local differentiation optimizes memory usage by storing only the essential information for each type of weight.
2Quantity of substance
If mask bits are generated and transmitted, then memory usage is reduced, but additional processing steps are required
Solution Approach 1:
The mask bits are generated in advance during the weight processing stage, before the actual neural network inference or training occurs. This preliminary generation of mask bits allows the reduced-weight representation to be stored efficiently, and the reconstruction process during inference simply involves reading the pre-generated mask bits and placing non-zero weights in their correct positions, rather than computing masks during critical inference paths.
Data Source
AI summary
Systems and methods for processing data for a neural network are described. The system comprises non-transitory memory configured to receive data bits defining a kernel of weights, the data bits being suitable for processing input data; and a data processing unit, configured to: receive bits defining a kernel of weights for the neural network, the kernel of weights comprising one or more non-zero value weights and one or more zero-valued weights; generate a set of mask bits, a position of each bit in the set of mask bits corresponds to a position within the kernel of weights and the value of each bit indicates whether a weight in the corresponding position is a zero-valued weight or a non-zero value weight; and transmit the non-zero value weights and the set of mask bits for storage, the non-zero value weights and the set of mask bits represent the kernel of weights.