Neural Network Training System Using Model Striping and Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training of deep neural networks (DNNs) for speech recognition is time-consuming due to high bandwidth costs associated with parallelization techniques, limiting their widespread use in recognition problems.
Innovation Solution
The implementation of data-parallel architectures using model striping, quantization, and efficient data transfer techniques to reduce the computational burden by processing gradient matrices across multiple processing units, allowing for faster training of DNNs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallelization techniques are used to train deep neural networks, then training speed is improved, but bandwidth costs increase
Solution Approach 1:
The model is divided into multiple stripes or partitions that can be processed independently across different processing units. Each processing unit handles a specific stripe of the model parameters, allowing parallel computation while reducing the amount of data that needs to be communicated between units. This segmentation enables faster training by distributing the computational load without requiring excessive bandwidth for data transfer.
Solution Approach 2:
The patent introduces a new dimension of parallelization by stripe index, in addition to the traditional data parallelism dimension. This allows the system to process different stripes of the model simultaneously across multiple processing units, effectively adding another layer of parallel computation that increases training speed without proportionally increasing bandwidth requirements.
2Measurement precision
If the size of updated model increases to process hundreds of millions of frames, then model accuracy is improved, but computation resource capacity is exceeded
Solution Approach 1:
The large model parameters are segmented into multiple stripes that can be processed and stored separately. This segmentation allows the system to handle large-scale models by distributing them across multiple processing units, each managing a manageable portion of the total model size. This approach enables processing of hundreds of millions of frames while keeping the computational burden on each individual resource unit within acceptable limits.
Solution Approach 2:
The system processes gradients and model updates in partial batches corresponding to different stripes, rather than requiring all model parameters to be updated simultaneously. This partial action approach allows the training process to proceed incrementally across different model stripes, reducing the peak memory and computation requirements while still achieving accurate results through multiple passes.
Data Source
AI summary
Techniques and constructs can reduce the time required to determine solutions to optimization problems such as training of neural networks. Modifications to a computational model can be determined by a plurality of nodes operating in parallel. Quantized modification values can be transmitted between the nodes to reduce the volume of data to be transferred. The quantized values can be as small as one bit each. Quantization-error values can be stored and used in quantizing subsequent modifications. The nodes can operate in parallel and overlap computation and data transfer to further reduce the time required to determine solutions. The quantized values can be partitioned and each node can aggregate values for a corresponding partition.