Hierarchical Token Retrieval for Long-Context LLM Coherence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face challenges in maintaining coherence and performing long-range reasoning when processing long documents, leading to performance drops, especially in domains like legal, engineering, and healthcare, and existing approaches like sliding window techniques and Retrieval-Augmented Generation (RAG) struggle with inefficient document chunking and loss of context.
Innovation Solution
The system employs multi-level generative AI models (h-LLMs) with iterative attention focusing (LASER) and context-optimized retrieval (SCORE-RAG) to refine and condense document context, using hierarchical tokens (H-Tokens) for enhanced coherence and semantic management, and a network of smaller LLMs for specialized tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs process long documents using traditional methods, then they can handle basic language tasks, but they fail to maintain coherence and perform long-range reasoning
Solution Approach 1:
The patent divides long documents into hierarchical segments (chunks) organized in a tree structure, where the root represents the entire document and child nodes represent progressively finer segments. This segmentation allows the LLM to process manageable portions while maintaining awareness of the overall document structure through the hierarchical relationships, thereby improving coherence and long-range reasoning across the full document length.
Solution Approach 2:
The patent implements a nested hierarchical structure where chunks are organized in multiple levels of containment, with larger chunks containing smaller sub-chunks. This nesting approach enables the system to maintain both high-level document context and detailed local information simultaneously, allowing the LLM to perform reasoning that spans across different scales of the document structure.
2Productivity
If sliding window techniques are used to process long documents, then the document can be divided into manageable segments, but context is lost at chunk boundaries
Solution Approach 1:
The patent pre-computes and stores hierarchical relationships, embeddings, and metadata for all chunks before processing queries. This preliminary action includes creating the chunk tree structure, calculating embeddings for each chunk at multiple hierarchical levels, and storing cross-chunk reference information. When a query is processed, this pre-computed context is readily available, eliminating the need to re-process entire documents and preventing context loss at boundaries.
Solution Approach 2:
The patent introduces hierarchical chunk embeddings and metadata as intermediaries that bridge the gap between individual chunks and the overall document context. These intermediaries capture semantic relationships across chunk boundaries and enable the LLM to retrieve relevant context from surrounding chunks without directly processing the entire document, thereby maintaining continuity across chunk boundaries.
3Adaptability or versatility
If Retrieval-Augmented Generation (RAG) is used, then external knowledge can be retrieved, but document chunking is inefficient and context is lost
Solution Approach 1:
The patent adds a hierarchical dimension to the traditional flat chunking approach by organizing chunks in a multi-level tree structure. Instead of simply dividing documents into linear segments, the system creates hierarchical groupings where chunks are nested within parent chunks and can be retrieved at multiple levels of granularity. This dimensional change enables more efficient retrieval by allowing the system to jump to relevant hierarchical levels rather than searching through all chunks linearly.
Solution Approach 2:
The patent changes the parameters of document representation by creating hierarchical embeddings at multiple levels (chunk-level, parent-chunk-level, document-level) rather than a single flat embedding. This parameter change allows the retrieval system to operate at different granularities and efficiently locate relevant context by comparing query embeddings against the hierarchical embedding structure, improving both efficiency and context preservation.
Data Source
AI summary
A method for utilizing hierarchical tokens (h-tokens) in large language models (LLMs) including receiving and processing an input prompt to generate tokens, identifying functional components derived from a subset of tokens, associating or generating h-tokens for at least one of the functional components by compressing the subset of tokens into an h-token for each functional component, associating or generating implementation characteristics including a defined function, an identified event, and/or an implementation method for each h-token, performing a retrieval augmented generation (RAG) processing function on each h-token, and generating a response from an output of the RAG processing function.