Hierarchical Document Chunking for Prompt-Limited RAG Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for chunking documents to fit within the prompt size limitations of generative AI models like GPT3.5-Turbo, GPT4, and GPT-4-32k result in low-quality matches and incorrect results due to the limitations of conventional chunking techniques, leading to a negative user experience.
Innovation Solution
A document hierarchy-based chunking process that utilizes the hierarchical structure of documents, such as the Document Object Model (DOM), to generate optimized chunks that fit within the prompt size limitations, enhancing the relevance and accuracy of matches by generating chunks based on headings and their associated content portions, and using embeddings for efficient retrieval augmented generation (RAG).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional chunking methods are used to break documents into smaller pieces, then the prompt size limitation is satisfied, but the match quality between user queries and content chunks deteriorates
Solution Approach 1:
The patent applies segmentation by dividing documents into chunks based on hierarchical structure (headings, subheadings, sections) rather than arbitrary character counts. This ensures each chunk represents a coherent semantic unit while fitting within prompt size limits, thereby maintaining both quantity constraints and match quality.
Solution Approach 2:
The patent introduces a hierarchical dimension to the chunking process by utilizing the document's existing structure (headings, subheadings, paragraphs). Instead of one-dimensional character-based splitting, the system multi-dimensionally segments documents along structural boundaries, improving semantic coherence and query matching while respecting prompt size constraints.
2Extent of automation
If documents are broken into smaller chunks to fit prompt limits, then the generative AI model can process the input, but the relevance and accuracy of the answers deteriorate
Solution Approach 1:
The patent applies preliminary action by pre-processing documents into hierarchically-structured chunks before they are fed to the generative AI model. This pre-chunking based on document structure ensures that when queries are processed, the relevant contextual information is already organized in semantically meaningful units, improving answer accuracy without compromising model processing capability.
3Quantity of substance
If arbitrary chunking is used to divide documents, then the prompt size constraint is met, but the user experience deteriorates due to irrelevant or incorrect results
Solution Approach 1:
The patent applies local quality by ensuring that each chunk has high semantic coherence and represents a distinct topical unit within the document hierarchy. By making each chunk locally meaningful and self-contained, the system improves relevance matching and user experience while maintaining compliance with prompt size constraints.
Data Source
AI summary
A method, computer program product, and computing system for identifying a plurality of headings from a document by processing a hierarchical structure associated with the document including the plurality of headings and a plurality of content portions within the plurality of headings. A plurality of respective chunks are generated using the plurality of headings and a prompt size limitation associated with a prompt of a generative artificial intelligence (AI) model. The plurality of respective chunks are provided for generating a prompt for the generative AI model.