DNN Memory Bandwidth Optimization via Incremental Weight Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) with many hidden layers and units are challenging to optimize in speech recognition systems, particularly on platforms with limited resources, as applying them to feature vectors every time frame can be computationally demanding and resource-intensive.
Innovation Solution
The method involves updating memory for selected output neurons of a DNN by computing incremental output values based on non-zero input neurons from previous layers, employing a batch setting to reduce memory access frequency, and using Single Instruction Multiple Data (SIMD) instructions to optimize memory bandwidth utilization, while skipping zero-input neurons and compressing weights to improve computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DNNs with many hidden layers and units are applied to feature vectors every time frame, then high-quality acoustic modeling is achieved, but computation time increases and resource consumption increases
Solution Approach 1:
The patent segments the DNN computation into multiple passes, where a first pass computes output values using compressed weights and a second pass refines them using un-compressed weights. This segmentation allows the system to maintain high modeling quality while reducing per-frame computation time by processing weights in batches rather than all at once.
Solution Approach 2:
The patent performs preliminary computation of output values using compressed weights before refining with un-compressed weights. This preliminary action reduces the computational burden during real-time processing by pre-processing weight data and preparing intermediate results that can be quickly refined.
2Reliability
If DNNs with many hidden layers and units are applied to feature vectors every time frame, then high-quality acoustic modeling is achieved, but resource consumption increases
Solution Approach 1:
The computation is divided into multiple passes with different weight precision requirements. The first pass uses compressed weights requiring fewer resources, while the second pass uses full-precision weights only where needed. This segmentation reduces overall resource consumption while maintaining the quality benefits of full-precision DNNs.
Solution Approach 2:
The patent changes the precision parameter of weights dynamically - using compressed representations during the first pass and un-compressed representations during the second pass. This parameter change allows the system to adapt resource usage to computational needs, reducing energy consumption while preserving accuracy.
3Loss of time
If memory access rate is reduced through batch setting, then computation time is improved, but memory bandwidth utilization becomes more challenging
Solution Approach 1:
The patent performs preliminary loading of weight data into memory before computation begins. By pre-positioning both compressed and un-compressed weights in memory, the system reduces the need for frequent memory accesses during computation, thereby reducing computation time while managing memory bandwidth through advance preparation.
Data Source
AI summary
Deep Neural Networks (DNNs) with many hidden layers and many units per layer are very flexible models with a very large number of parameters. As such, DNNs are challenging to optimize. To achieve real-time computation, embodiments disclosed herein enable fast DNN feature transformation via optimized memory bandwidth utilization. To optimize memory bandwidth utilization, a rate of accessing memory may be reduced based on a batch setting. A memory, corresponding to a selected given output neuron of a current layer of the DNN, may be updated with an incremental output value computed for the selected given output neuron as a function of input values of a selected few non-zero input neurons of a previous layer of the DNN in combination with weights between the selected few non-zero input neurons and the selected given output neuron, wherein a number of the selected few corresponds to the batch setting.