Hierarchical Sparse Matrix Representation for Neural Network Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional sparsity representations in neural networks lead to inefficient storage and computation due to large memory usage and unnecessary operations on zero elements in sparse matrices, which are not hardware-friendly for decompression.
Innovation Solution
A hierarchical representation of sparse matrices using a first level bitmap, a second level bitmap, and an element array, where the first level bitmap indicates block non-zero elements, and the second level bitmap provides detailed non-zero element information only when needed, allowing for efficient decompression and execution within the neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sparse matrices are stored in conventional format, then storage space is allocated, but memory usage is excessively large due to storing zeros
Solution Approach 1:
The patent extracts only the non-zero elements from the sparse matrix and stores them in a compressed format with bitmap indicators. This removes the unnecessary zero elements from storage, directly resolving the contradiction between allocating storage space and wasting it on zero elements.
Solution Approach 2:
The patent changes the storage parameter from storing all matrix elements (including zeros) to storing only non-zero elements with their position indicators. This parameter change in the storage format reduces memory usage while preserving the essential information needed for neural network computation.
2Productivity
If conventional sparse matrix representation is used, then computation can be performed, but unnecessary operations on zero elements increase computation time
Solution Approach 1:
The patent extracts and processes only the non-zero elements during computation operations. By removing zero elements from the computation pipeline through the bitmap-based compressed format, the system performs fewer unnecessary operations, directly improving computation throughput and reducing time loss.
3Quantity of substance
If sparse matrix decompression is implemented, then storage efficiency is improved, but decompression complexity increases
Solution Approach 1:
The patent segments the decompression process into two simple stages: first reading the bitmap to identify non-zero element positions, then reading the corresponding non-zero values. This segmentation simplifies the decompression logic compared to conventional approaches, resolving the contradiction between storage efficiency and decompression complexity.
Solution Approach 2:
The patent performs preliminary organization of data during the encoding phase by creating the bitmap structure that marks non-zero element positions. This preliminary action simplifies the subsequent decompression process, as the position information is already prepared and does not require complex computation during decompression.
4Quantity of substance
If hierarchical bitmap representation is used, then memory storage is reduced, but data structure complexity increases
Solution Approach 1:
The patent segments the sparse matrix data structure into two distinct components: a bitmap layer for position identification and a values layer for storing non-zero elements. This segmentation creates a clear, hierarchical structure that reduces memory usage while maintaining manageable complexity through organized data separation.
Data Source
AI summary
The present disclosure relates to a method and an apparatus for representation of a sparse matrix in a neural network. In some embodiments, an exemplary operation unit includes a buffer for storing a representation of a sparse matrix in a neural network, a sparse engine communicatively coupled with the buffer, and a processing array communicatively coupled with the sparse engine. The sparse engine includes circuitry to: read the representation of the sparse matrix from the buffer, the representation comprising a first level bitmap, a second level bitmap, and an element array; decompress the first level bitmap to determine whether a block of the sparse matrix comprises a non-zero element; and in response to the block comprising a non-zero element, decompress the second level bitmap using the element array to obtain the block of the sparse matrix. The processing array includes circuitry to execute the neural network with the sparse matrix.