Non-autoregressive Transformer Positional Encoding and Masked Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-autoregressive transformer models struggle with effectively characterizing positional information and often generate repetitive output tokens, requiring distillation from autoregressive models to achieve comparable performance, and infrequently learn from all-masked sequences during inference.
Innovation Solution
The proposed transformer model improves positional representations through a learned position combination layer and incorporates a masked attention layer to maintain order awareness, while training with an additional loss component to correct predictions made during initial decoding iterations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If non-autoregressive models generate multiple output tokens in parallel to accelerate inference, then productivity is improved, but manufacturing precision deteriorates due to ineffective positional information characterization and repetitive token generation
Solution Approach 1:
The model performs preliminary actions by generating initial output token estimates in parallel before refinement. The decoder iteratively refines these estimates, using the initial parallel generation as a starting point that can be progressively improved through multiple decoding iterations, combining the benefits of fast parallel initialization with accurate sequential refinement
Solution Approach 2:
The model implements feedback mechanisms where the decoder's output estimates are fed back into subsequent decoding iterations. The model uses its own predictions as input for refinement, allowing errors from initial parallel generation to be corrected in later iterations while maintaining the acceleration benefit of the initial parallel step
2Adaptability or versatility
If the decoder is applied iteratively with all-masked output estimates during inference, then adaptability is improved, but manufacturing precision deteriorates because the model infrequently learns from all-masked sequences during training
Solution Approach 1:
The model inverts the typical training approach by training with all-masked output sequences instead of partially-masked sequences. This inversion allows the model to learn directly from the inference condition (all-masked) during training, eliminating the distribution mismatch between training and inference and improving accuracy for all-masked inputs
3Measurement precision
If positional encodings are added to token embeddings to characterize position information, then measurement precision is improved, but device complexity increases due to additional encoding layers
Solution Approach 1:
The model merges positional encodings with token embeddings into a unified representation. Instead of treating them as separate components that increase complexity, the positional information is integrated directly into the token embeddings, creating a combined representation that maintains positional accuracy while simplifying the overall architecture
Data Source
AI summary
An autoencoder model includes an encoder portion and a decoder portion. The encoder encodes an input token sequence to an input sequence representation that is decoded by the decoder to generate an output token sequence. The autoencoder model may decode multiple output tokens in parallel, such that the decoder may be applied iteratively. The decoder may receive an output estimate from a prior iteration to predict output tokens. To improve positional representation and reduce positional errors and repetitive tokens, the autoencoder may include a trained layer for combining token embeddings with positional encodings. In addition, the model may be trained with a corrective loss based on output predictions when the model receives a masked input as the output estimate.