Online Attention Sequence Processing for Real-Time Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional sequence processing systems using attention mechanisms require high computational resources and cannot generate outputs until the entire input sequence is received, limiting their applicability in real-time applications like speech recognition and machine translation.
Innovation Solution
A novel attention mechanism that allows the decoder to start generating outputs before the entire input sequence is processed, using a decoder recurrent neural network with a context vector selection engine that enforces monotonicity and employs a differentiable training process to reduce computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional attention mechanisms are used in sequence processing systems, then the system can process sequences with high accuracy, but the computational resources required become excessively high and the system cannot generate outputs until the entire input sequence is received
Solution Approach 1:
The patent segments the attention computation process by introducing a selection engine that chooses only the most relevant encoded representation (context vector) from the sequence, rather than computing attention scores with all encoded representations. This segmentation reduces computational complexity from O(TU) to O(max(T,U)) while maintaining processing accuracy through the monotonicity constraint that ensures the selected context vector is appropriate for each output time step
2Measurement precision
If the system waits for the entire input sequence to be processed before generating outputs, then it can use complete information for accurate processing, but real-time processing capability is lost
Solution Approach 1:
The patent enables preliminary action by allowing the decoder to start generating outputs before the encoder finishes processing the entire input sequence. The selection engine can identify and select the appropriate context vector from already-processed encoded representations, enabling online/real-time processing without sacrificing accuracy through the monotonicity constraint that ensures proper context selection
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence including a respective output at each of multiple output time steps from respective encoded representations of inputs in an input sequence. The method includes, for each output time step, starting from the position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, traversing the encoded representations until an encoded representation is selected as a current context vector at the output time step. A decoder neural network processes the current context vector and a preceding output at the preceding output time step to generate a respective output score for each possible output and to update the hidden state of the decoder recurrent neural network. An output is selected for the output time step using the output scores.