Hierarchical Retrieval Units for LLM Context Window Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face limitations due to a limited context window, leading to computational inefficiencies and loss of information when processing large documents, especially when dealing with multimodal content like tables and non-textual elements, which existing chunking techniques often handle inadequately, resulting in incomplete context and fidelity issues.
Innovation Solution
A lossless hierarchical regurgitation technique transforms documents into a structured hierarchical representation, preserving all information and organizing it into manageable retrieval units, allowing LLMs to access and utilize information with greater completeness and structural integrity, thereby overcoming context window limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If LLMs process larger context windows to handle more document information, then information completeness improves, but computational load and energy consumption increase excessively
Solution Approach 1:
The patent divides the document into multiple hierarchical levels (document → sections → subsections → paragraphs → sentences). This segmentation allows the system to process and retrieve only relevant portions rather than the entire document, reducing computational load while maintaining information completeness. The hierarchical structure enables selective access to specific segments based on query relevance.
Solution Approach 2:
The patent implements nested hierarchical levels where sections contain subsections, which contain paragraphs, which contain sentences. Each level is embedded within the previous level, creating a nested structure. This allows the system to navigate from coarse-grained sections to fine-grained sentences efficiently, processing only the necessary depth required for each query, thereby reducing overall computational requirements.
2Productivity
If traditional chunking methods divide documents into smaller units, then processing efficiency improves, but semantic coherence and contextual structure are disrupted
Solution Approach 1:
The patent segments documents using semantically meaningful boundaries (sections, subsections, paragraphs, sentences) rather than arbitrary fixed-size chunks. This segmentation preserves the natural hierarchical structure and semantic coherence of the original document while creating manageable units for efficient processing and retrieval.
Solution Approach 2:
The patent introduces a hierarchical dimension to the chunking process, organizing segments into multiple levels (document → sections → subsections → paragraphs → sentences). This hierarchical dimension allows the system to maintain semantic coherence within each level while enabling efficient processing by navigating through the hierarchy, thus resolving the conflict between processing efficiency and semantic preservation.
3Productivity
If summarization techniques are used to compress document content, then context length requirements reduce, but fidelity and exact recall of facts are lost
Solution Approach 1:
The patent segments the document into hierarchical units without summarizing or compressing the content. Each segment (section, subsection, paragraph, sentence) retains the original text verbatim, ensuring complete fidelity and exact recall capability. The segmentation alone provides the compression effect by enabling selective retrieval of only relevant segments rather than processing the entire document.
4Quantity of substance
If hierarchical structures are built with compressed representations, then storage requirements reduce, but critical details are lost
Solution Approach 1:
The patent segments the document into hierarchical units and stores each segment as a discrete unit with its own metadata (level, position, parent-child relationships). This segmentation allows the system to store only the necessary hierarchical structure information rather than compressed representations, preserving all critical details while achieving storage efficiency through selective retrieval of relevant segments based on query needs.
Data Source
AI summary
A document preparation method involves creating a hierarchical representation of an input document without summarizing or omitting any content. The method uses a generative language model to generate the hierarchical representation and stores it in a repository for later use by a client generative language model. This allows for more accurate and complete generation of text, enabling the use of retrieval units to enhance the output of the client generative language model while efficiently exploiting its limited context window.