Long-Prompt Generative Models With Chunked Attention Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional generative machine learning models struggle with processing long input sequences due to memory constraints, computational complexity, and accuracy losses when the input length differs from the sequence length used during training.
Innovation Solution
Implementing chunk-based iterative compression and cognitive contextual retention techniques, including dynamic compression of intermediate data and reorganization of positional embeddings, to process longer input sequences efficiently while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If attention operations are used to process long input sequences, then accuracy and reliability of output predictions are improved, but memory usage increases and computational complexity increases quadratically
Solution Approach 1:
The patent divides the long input sequence into multiple chunks or segments that can be processed separately. Each chunk is processed through the attention mechanism independently, and the results are then aggregated. This segmentation approach allows the model to maintain high accuracy for each segment while avoiding the quadratic memory growth that would occur if the entire long sequence were processed at once.
Solution Approach 2:
The patent introduces a temporal dimension by processing sequences in multiple passes or stages. Instead of processing the entire sequence in a single forward pass, the model processes chunks sequentially over time, retaining only essential information from previous chunks. This transforms the problem from a single high-dimensional memory burden into a series of lower-dimensional processing steps.
2Reliability
If attention operations are used to process long input sequences, then accuracy and reliability of output predictions are improved, but computational complexity increases quadratically
Solution Approach 1:
The patent segments the computational workload by dividing the long sequence into chunks that are processed separately. Each chunk requires attention operations only with other chunks or with a compressed representation, reducing the quadratic complexity from O(n²) for the entire sequence to O((n/k)² × k) where k is the number of chunks, which simplifies to O(n²/k) - a significant reduction when k > 1.
Solution Approach 2:
The patent extracts and retains only the most important or representative information from each chunk, discarding redundant details. This extraction process creates a compressed representation that captures essential patterns while removing unnecessary computational burden, allowing the model to maintain accuracy on critical information while reducing overall computational complexity.
3Ease of manufacture
If fixed sequence length is used during training, then model training is simplified, but accuracy is lost when input length differs from training sequence length
Solution Approach 1:
The patent implements dynamic chunking that adapts to the actual input length. Instead of forcing all inputs to a fixed length, the system dynamically determines optimal chunk sizes and numbers based on the input being processed. This dynamic approach maintains the simplicity of fixed-length processing during training while enabling flexible handling of variable-length inputs during inference, preserving accuracy across different input sizes.
Solution Approach 2:
The patent changes the parameter of sequence length from a fixed constant to a variable that can adapt to input requirements. By using configurable chunk sizes and numbers, the system can adjust its processing parameters based on the specific input, maintaining training simplicity while achieving accuracy on variable-length inputs through parameter flexibility rather than architectural complexity.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for machine learning. In an example method, a set of data is generated based on a subset of tokens, from a sequence of tokens used as an input prompt to a generative machine learning model, using an attention mechanism of the generative machine learning model. The set of data is compressed based on a respective novelty score of each respective token of the first subset of tokens in accordance with one or more memory criteria. A set of positional embeddings associated with the compressed set of data is reorganized, and an output of the generative machine learning model is generated based on the compressed set of data and the reorganized set of positional embeddings.