Task Attention Mechanism for Ultra-Long LLM Contexts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face a context window limitation that discards text exceeding their maximum token length, leading to information loss and reduced processing efficiency.
Innovation Solution
A task attention mechanism and virtual memory graph are employed to convert ultra-long text into a knowledge graph, splitting it into subtexts and subgraphs, which are combined to form a searchable logical graph, enhancing memory retrieval and logical reasoning capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If LLMs process text data exceeding their maximum token length, then information completeness is improved, but processing efficiency deteriorates due to context window limitations
Solution Approach 1:
The patent divides ultra-long text data into multiple subtexts, each within the context window limit, and processes them separately to form subgraphs that are then combined into a comprehensive logical graph, resolving the contradiction between information completeness and processing efficiency
Solution Approach 2:
The patent transforms linear text data into a graph structure representation, enabling the model to access and process information beyond the traditional sequential context window limitation, thus improving information completeness without sacrificing processing efficiency
2Productivity
If LLMs truncate text to fit context window, then processing efficiency is improved, but information loss increases
Solution Approach 1:
The patent performs preliminary processing by converting text data into a graph structure before main processing, organizing information in a way that preserves completeness while enabling efficient retrieval and processing during inference
Solution Approach 2:
The patent introduces a graph structure as an intermediary representation between the original text and the LLM processing, allowing the model to access comprehensive information without directly processing the full ultra-long text sequence
3Reliability
If LLMs use attention mechanism on ultra-long text, then context understanding is improved, but computational complexity increases
Solution Approach 1:
The patent segments ultra-long text into subtexts processed separately, with each segment generating a subgraph that captures local context relationships, reducing the computational burden compared to processing the entire text sequence at once while maintaining overall context understanding
Solution Approach 2:
The patent combines multiple subgraphs into a unified logical graph that preserves contextual relationships across all segments, enabling comprehensive context understanding through structured integration rather than direct attention on the full ultra-long text
Data Source
AI summary
One or more computer-implemented methods, computer systems and/or computer program products of use provided herein relate to employing a task attention mechanism for context window augmentation of a large language model (LLM). In various embodiments, a computer-implemented method comprises receiving, at a machine learning model, first text data. The computer-implemented method further comprises determining, via the machine learning model, a token length of the first text data. The computer-implemented method further comprises in response to determining that the token length is greater than a threshold token length of a first language machine learning model, generating via the machine learning model a first logical graph from the first text data, wherein the first logical graph incorporates tokens exceeding the threshold token length of the first language machine learning model without a portion of the first text data becoming eliminated, and the first logical graph is incorporated within the machine learning model.