Storage Class Memory Zero-Index Skip for Sparse CNN Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face computational intensity and high energy consumption during inference operations due to the need for extensive matrix multiplication using sparse filters, where many operations involve zero-valued weights, leading to inefficiencies in data access and processing.
Innovation Solution
Implementing a zero-column index (ZCI) and zero-row index (ZRI) mechanism in storage class memory arrays to skip accessing all-zero columns and rows, allowing for in-memory matrix multiplication operations that reduce unnecessary computations and data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If in-memory matrix multiplication is performed using storage class memory arrays, then computational speed is improved, but energy consumption increases due to accessing all columns and rows including those with zero-valued weights
Solution Approach 1:
The patent extracts and identifies zero columns and zero rows from the weight matrix during preprocessing. These zero-valued elements are removed from the computation by storing their positions in index structures, allowing the in-memory computation to skip accessing them entirely, thus reducing energy consumption while maintaining computational speed.
Solution Approach 2:
The patent performs preliminary identification and marking of zero columns and zero rows before the actual matrix multiplication operation. By pre-computing and storing the indices of zero-valued elements, the system prepares the data structures in advance so that during inference, unnecessary memory accesses are avoided, reducing energy consumption without sacrificing speed.
2Ease of operation
If traditional matrix multiplication is performed without skipping zero-valued weights, then implementation simplicity is maintained, but computational efficiency deteriorates due to unnecessary operations
Solution Approach 1:
The patent segments the weight matrix into non-zero elements and zero elements by creating separate index structures for zero columns and zero rows. This segmentation allows the computation to be divided into necessary operations (accessing only non-zero elements) and skipped operations, improving computational efficiency while maintaining relatively simple implementation through the use of index arrays.
Solution Approach 2:
The patent introduces index structures (zero column indices and zero row indices) as intermediary data structures that mediate between the weight matrix and the computation process. These indices act as a bridge that guides the computation to skip zero-valued elements, improving efficiency without significantly complicating the overall implementation.
3Reliability
If all columns and rows are accessed during matrix multiplication, then data access completeness is ensured, but data transfer volume increases leading to higher energy consumption
Solution Approach 1:
The patent extracts the positions of zero columns and zero rows into separate index structures, allowing the system to take these unnecessary data access operations out of the computation flow. By identifying and removing references to zero-valued elements, the system maintains data access completeness for necessary elements while eliminating energy-wasting accesses to zero elements.
Solution Approach 2:
The patent performs preliminary identification of zero columns and rows, storing their indices in advance. This preliminary action enables the system to know beforehand which columns and rows should be skipped, ensuring that data access is complete for all necessary elements while avoiding unnecessary transfers of zero-valued data, thus reducing energy waste.
Data Source
AI summary
Techniques are presented for accelerating in-memory matrix multiplication operations for a convolution neural network (CNN) inference in which the weights of a filter are stored in the memory of a storage class memory device, such as a ReRAM or phase change memory based device. To improve performance for inference operations when filters exhibit sparsity, a zero column index and a zero row index are introduced to account for columns and rows having all zero weight values. These indices can be saved in a register on the memory device and when performing a column/row oriented matrix multiplication, if the zero row/column index indicates that the column/row contains all zero weights, the access of the corresponding bit/word line is skipped as the result will be zero regardless of the input.