RNN-T Training With Selective Probability Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural transducer models for automatic speech recognition (ASR) face challenges in managing high memory requirements during training, particularly due to the large output vocabulary size, which limits training batch size and slows down the training process.
Innovation Solution
The proposed solution involves modifying the neural transducer model to include a fused joint network with a fused linear and softmax layer, and dividing the computation of joint embeddings into slices during training. This approach reduces the memory storage requirements by only storing and recalculating necessary token outputs, such as the next blank and next token, during the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the output vocabulary size is increased to improve language coverage, then the model's adaptability improves, but the memory storage requirements increase significantly
Solution Approach 1:
The patent extracts and stores only the necessary probability outputs (next blank and next token) during the forward pass, while discarding the full vocabulary probability distributions. This selective extraction reduces memory storage requirements from storing all K vocabulary probabilities to storing only 2 probabilities per time step, enabling larger vocabulary sizes without proportional memory increases.
Solution Approach 2:
The patent segments the backpropagation process by dividing the joint embeddings computation into slices, processing only the necessary portions during backpropagation rather than requiring full vocabulary outputs at all times. This segmentation allows the system to handle large vocabularies by processing data in manageable chunks.
2Productivity
If the training batch size is increased to improve training efficiency, then the productivity improves, but the memory storage requirements increase
Solution Approach 1:
By extracting and storing only the necessary probability outputs (next blank and next token) rather than all vocabulary probabilities, the patent enables larger training batch sizes to be stored in memory. This selective storage allows the system to increase batch size for improved training efficiency without the memory burden of storing complete vocabulary distributions for all tokens.
3Adaptability or versatility
If the memory capacity is increased to support larger vocabulary, then the adaptability improves, but the computational cost increases
Solution Approach 1:
The patent extracts and stores only the necessary probability values (next blank and next token) during the forward pass, reducing the amount of data that needs to be stored and processed during backpropagation. This selective extraction reduces the computational cost of handling large vocabularies by working with only the essential probability information rather than complete vocabulary distributions.
Solution Approach 2:
The patent segments the computation by dividing joint embeddings into slices and processing only necessary portions during backpropagation. This segmentation reduces the computational burden when handling large vocabularies by processing data in manageable chunks rather than requiring full vocabulary processing at all times.
Data Source
AI summary
Efficient training is provided for models comprising RNN-T (recurrent neural network transducers). The model transducers comprise an encoder, a decoder, and a fused joint network. The fused joint network receives encoding and decoding embeddings from the encoder and decoder. During training, the model stores the probability data for the next blank output and the next token at each time step rather than storing all probabilities for all possible outputs. This can significantly reduce requirements for memory storage, while still preserving the relevant information required to calculate the loss that will be backpropagated through the neural transducer during training to update the parameters of the neural transducer and to generate a trained or modified neural transducer. The computation of embeddings can also be divided into small slices and some of the utterance padding used for the training samples can also be removed to further reduce the memory storage requirements.