Neural Network Training via Optimal Completion Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training techniques for sequence generation, such as Maximum Likelihood Estimation (MLE) and scheduled sampling, require excessive computational resources and do not achieve state-of-the-art performance in tasks like speech recognition.
Innovation Solution
The use of optimal completion distillation (OCD) to train neural networks by generating output sequences and applying quality scores based on prefixes relative to ground truth sequences, reducing computational resources and enhancing performance without pre-training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If Maximum Likelihood Estimation (MLE) is used to train neural networks for sequence generation, then the training process is simple and straightforward, but the computational resources required become excessive and performance does not reach state-of-the-art levels
Solution Approach 1:
The patent extracts only the necessary computational operations from the full MLE training process. Instead of training on complete ground truth sequences, it extracts and uses only prefix segments of length K from the ground truth, significantly reducing the computational burden while maintaining training effectiveness
Solution Approach 2:
The patent applies partial action by training the neural network on partial sequences (prefixes of length K) rather than complete sequences. This partial training approach reduces computational resources required while achieving state-of-the-art performance, demonstrating that full MLE training is excessive in this context
2Ease of operation
If conventional training techniques are used, then the training process is straightforward, but the neural network requires pre-training and consumes excessive computational resources
Solution Approach 1:
The patent eliminates the need for pre-training by incorporating the preliminary action of generating candidate sequences and computing quality scores directly into the main training loop. The neural network is trained end-to-end from scratch using the proposed OCD method, removing the complex pre-training stage while maintaining operational ease
3Use of energy by moving object
If quality scores based on prefixes are used to train neural networks, then computational resources are reduced and state-of-the-art performance is achieved, but the training method becomes more complex
Solution Approach 1:
The patent introduces quality scores as an intermediary mechanism that bridges the gap between reduced computational complexity and state-of-the-art performance. These quality scores, computed based on prefix matches against ground truth, serve as a mediator that enables efficient training without requiring excessive computational resources
Solution Approach 2:
The patent changes the training parameters by introducing a prefix length parameter K and using quality scores based on prefix matching instead of complete sequence likelihood. This parameter change enables the model to achieve state-of-the-art performance with reduced computational resources, balancing complexity and efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a sequence generation neural network. One of the methods includes obtaining a batch of training examples; for each of the training examples: processing the training network input in the training example using the neural network to generate an output sequence; for each particular output position in the output sequence: identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence, for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output, and determining an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score.