Non-autoregressive Decoder Distillation for Iteration Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Iterative non-autoregressive decoder models require multiple iterations to achieve accurate output token predictions, leading to increased computation time and potential loss of time-related advantages over sequential autoregressive translation models, while also producing repeated tokens and other accuracy challenges.
Innovation Solution
A teacher-student distillation approach is employed, where a pre-trained iterative non-autoregressive transformer serves as a teacher to guide a student model in reducing the number of iterations needed for output token prediction, using a shared architecture and a loss function based on prediction differences and hidden states to improve the student's parameter updates and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If iterative non-autoregressive decoding is applied multiple times to improve output accuracy, then manufacturing precision is improved, but loss of time increases
Solution Approach 1:
The teacher model performs multiple iterations of decoding in advance during the distillation process, generating soft labels that encode the optimal multi-iteration results. The student model learns from these pre-computed results, achieving high accuracy without performing multiple iterations itself, thus saving time while maintaining precision
Solution Approach 2:
The student model copies the knowledge and decision-making patterns of the teacher model through distillation, learning to make accurate predictions in a single iteration by mimicking the teacher's multi-iteration output distribution, thereby transferring accuracy without replicating the iterative process
2Manufacturing precision
If the number of decoder iterations is increased to reduce repeated tokens, then manufacturing precision is improved, but productivity decreases
Solution Approach 1:
The teacher model预先 performs multiple iterations to identify and resolve repeated tokens, embedding this knowledge in the soft labels. The student model inherits this capability through distillation, achieving token diversity and accuracy without performing multiple iterations, thus maintaining productivity while improving precision
3Loss of time
If teacher-student distillation is applied to reduce iterations, then loss of time is reduced, but device complexity increases
Solution Approach 1:
The soft labels generated by the teacher model serve as an intermediary that bridges the gap between multi-iteration accuracy and single-iteration speed. This intermediary contains the distilled knowledge, allowing the student model to achieve high performance without directly implementing the complex multi-iteration process, thus reducing computational time while managing complexity through the intermediary representation
Data Source
AI summary
A non-autoregressive transformer model is improved to maintain output quality while reducing a number of iterative applications of the model by training parameters of a student model based on a teacher model. The teacher model is applied several iterations to a masked output and a student model is applied one iteration, such that the respective output token predictions for the masked positions can be compared and a loss propagated to the student. The loss may be based on token distributions rather than the specific output tokens alone, and may additionally consider hidden state losses. The teacher model may also be updated for use in further training based on the updated model, for example, by updating its parameters as a moving average.