LLM Token Selection Inference for Long-Context Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based language models (LLMs) are computationally expensive and memory-intensive due to their numerous transformer decoder layers, requiring significant resources for processing large contexts.
Innovation Solution
An LLM inference framework that performs inference on a small subset of input tokens by selecting tokens receiving the most attention from early transformer layers, reducing the context length for subsequent processing and using only a portion of the transformer layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all transformer decoder layers process all input tokens in large contexts, then model accuracy is maintained, but memory consumption and computational cost increase significantly
Solution Approach 1:
The patent segments the processing of input tokens by dividing them into two categories: selected tokens (processed through all transformer layers) and non-selected tokens (processed only through early transformer layers). This segmentation allows the system to maintain model accuracy for important tokens while reducing overall computational load and memory consumption for large contexts.
Solution Approach 2:
The patent applies local quality by differentiating the processing depth for different tokens based on their importance. Selected tokens receive full processing through all transformer layers to maintain high accuracy, while non-selected tokens receive reduced processing through only early layers, optimizing the balance between accuracy and resource consumption.
2Measurement precision
If all transformer decoder layers process all input tokens in large contexts, then model accuracy is maintained, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the computational workload by processing different tokens through different numbers of layers. Non-selected tokens are processed only through early transformer layers, significantly reducing computational cost and improving processing speed, while selected tokens receive full processing to maintain accuracy.
Solution Approach 2:
The patent applies partial action by processing non-selected tokens through only a portion of the transformer layers (early layers) rather than all layers. This partial processing reduces computational cost and improves productivity while the selected tokens receive complete processing to maintain model accuracy.
3Productivity
If a subset of tokens is selected for full processing, then computational efficiency improves, but model accuracy may deteriorate
Solution Approach 1:
The patent applies preliminary action by using early transformer layers to pre-process all input tokens and identify which tokens are most important (selected tokens) before proceeding to full processing. This preliminary selection ensures that the most critical tokens are identified for complete processing, maintaining model accuracy while improving computational efficiency.
Solution Approach 2:
The patent uses feedback from the early transformer layer processing to inform the selection of tokens for full processing. The attention mechanisms and intermediate representations from early layers provide feedback about token importance, enabling the system to select the most relevant tokens for complete processing through all transformer layers.
Data Source
AI summary
Embodiments described herein provide A method for generating a response to an input context by a neural network based language model (LM) with a plurality of neural network layers, comprising: converting the input context into a plurality of tokens; generating one or more intermediate values associated with each of the plurality of tokens utilizing a subset of the plurality of neural network layers; selecting a subset of the plurality of tokens having highest associated intermediate values; and generating, based on the subset of the plurality of tokens, the response utilizing all of the plurality of neural network layers of the LM.