Sparse Neural Network Data Compaction for Memory Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks, face significant computational and energy efficiency challenges during inference, especially on mobile platforms like autonomous vehicles and cameras, due to large computation requirements and high energy consumption, which existing technologies have not adequately addressed.
Innovation Solution
A deep learning accelerator architecture that employs data compaction and zero gating techniques to reduce computational requirements by identifying and eliminating zero values in neural network weights and activations, using a compaction engine to transmit single-bit signals for zeros and an expansion engine to generate expanded data sequences for processing elements, thereby reducing energy consumption and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-precision multi-bit data is transmitted for all neural network parameters, then computational accuracy is maintained, but memory bandwidth consumption and energy usage increase significantly
Solution Approach 1:
The patent extracts and removes zero values from the data stream, transmitting only non-zero values along with their position information. This extraction of unnecessary zero data significantly reduces the amount of data that needs to be transmitted through memory bandwidth, directly addressing the contradiction between maintaining computational accuracy and reducing memory bandwidth consumption.
Solution Approach 2:
The patent changes the data representation parameter from full-precision multi-bit values to a compact format consisting of position markers and reduced-precision values. This parameter transformation allows the system to maintain sufficient computational accuracy while dramatically reducing the data size and associated memory bandwidth requirements.
2Reliability
If neural network layers are made deeper and input sets are enlarged to improve model capability, then classification performance increases, but computational requirements and energy consumption become prohibitively large
Solution Approach 1:
By extracting and removing zero values from neural network activations and weights, the patent reduces the effective computational workload. This allows deeper networks with more layers and larger input sets to achieve improved classification performance while consuming significantly less energy, as the system only processes non-zero values through the computational graph.
Solution Approach 2:
The patent applies partial action by processing only the necessary non-zero portions of data rather than all data uniformly. This selective processing approach enables the system to handle deeper networks and larger inputs that would otherwise be computationally prohibitive, achieving improved performance without excessive energy consumption.
3Use of energy by stationary object
If data compaction is applied to reduce bandwidth and energy usage, then memory bandwidth consumption decreases significantly, but data processing complexity increases due to compaction and expansion operations
Solution Approach 1:
The patent introduces compaction and expansion engines as intermediary components that bridge the gap between memory storage and computational processing. These intermediaries handle the data format transformation, allowing the rest of the system to operate with simplified data handling while the intermediaries manage the complexity of compaction and expansion operations.
Data Source
AI summary
A method, computer program product, and system for sparse convolutional neural networks that improves efficiency is described. Multi-bit data for input to a processing element is received at a compaction engine. The multi-bit data is determined to equal zero and a single bit signal is transmitted from the memory interface to the processing element in lieu of the multi-bit data, where the single bit signal indicates that the multi-bit data equals zero. A compacted data sequence for input to a processing element is received by a memory interface. The compacted data sequence is transmitted from the memory interface to an expansion engine. Non-zero values are extracted from the compacted data sequence and zeros are inserted between the non-zero values by the expansion engine to generate an expanded data sequence that is output to the processing element.