Online Sequence Generation via Progress Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network systems require processing the entire input sequence before generating outputs, leading to high latency and increased computational resources in tasks like voice recognition and translation, and unstable training due to high variance in parameter updates.
Innovation Solution
Implementing an attention-based sequence-to-sequence processing system that generates outputs online by using progress scores to determine when to emit outputs at each time step, reducing the need for attention over all inputs and employing variational inference with a multi-sample objective to stabilize training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional attention mechanism is applied over all inputs in the input sequence, then the accuracy of sequence-to-sequence tasks is improved, but the latency between receiving the first input and generating the final output increases significantly
Solution Approach 1:
The patent segments the attention mechanism into progressive stages, where attention is applied to subsets of the input sequence rather than the entire sequence at once. This allows the system to generate intermediate outputs before processing all inputs, thereby reducing latency while maintaining accuracy through staged attention applications.
Solution Approach 2:
The patent implements preliminary processing of input sequences by precomputing attention weights for portions of the input before the entire sequence is available. This preliminary action enables the system to begin generating outputs early, reducing overall latency while preserving accuracy through subsequent refinement as more inputs become available.
2Measurement precision
If conventional attention mechanism processes the entire input sequence before generating outputs, then the accuracy of tasks like voice recognition and translation is improved, but the computational resources required increase
Solution Approach 1:
The patent divides the computational workload into segments by applying attention mechanisms to portions of the input sequence separately and progressively. This segmentation reduces the peak computational resource requirements compared to processing the entire sequence simultaneously, while maintaining accuracy through cumulative attention applications.
Solution Approach 2:
The patent implements partial attention processing where the system applies attention to only the necessary portions of the input sequence at each processing stage rather than the entire sequence. This partial action reduces computational resource consumption while achieving sufficient accuracy for the task at hand.
3Loss of information
If the system waits for the entire input sequence to be processed before generating outputs, then the attention mechanism can operate over complete information, but the productivity of the system decreases
Solution Approach 1:
The patent performs preliminary attention processing on available input portions before the complete input sequence is received. This preliminary action enables early output generation while preserving the option to refine results as more input information becomes available, thereby maintaining productivity without sacrificing information completeness.
Solution Approach 2:
The patent implements continuous output generation throughout the processing of the input sequence rather than waiting for completion. This continuity maintains productivity by generating outputs as soon as sufficient information is available, while the system continues to process remaining inputs to refine and complete the output sequence.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence from a source sequence. In one aspect, the system includes a recurrent neural network configured to, at each time step, receive an input for the time step and process the input to generate a progress score and a set of output scores; and a subsystem configured to, at each time step, generate the recurrent neural network input and provide the input to the recurrent neural network; determine, from the progress score, whether or not to emit a new output at the time step; and, in response to determining to emit a new output, select an output using the output scores and emit the selected output as the output at a next position in the output order.